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ABSTRACT 

The goal of the Job Training Partnership Act (JTPA) 
is long-term employability . To obtain feedback on the effectiveness 
of JTPA programs in promoting retention in the labor force, the U.S. 
Department of Labor (DDL) requires states and service delivery areas 
(SDAs) to collect information on the postprogram outcomes of JTPA 
participants. This technical assistance guide helps states collect 
data. This guide helps all those involved in the JTPA follow-up 
process to obtain accurate, valid data that meet the federal 
reporting requirements, provide an equitable basis for incentive 
awards, and yield useful information for program management decisions 
at all levels. In addition, it presents a detailed description of 
follow-up and validation strategies gathered in a recent study 
conducted for the DOL. This study surveyed follow-up staff in 50 
states and the District of Columbia and Puerto Rico, plus 151 SDAs 
were surveyed to determine how follow-ups are conducted, how accuracy 
of data is ensured, and what strategies are used for follow-up data 
collection. The guide is intended to help users adapt existing 
procedures or adopt new strategies to improve information obtained. 
Eight references, 8 appendices, and 21 exhibits are included in this 
report. The appendices include: (1) JTPA Annual Status Report and 
Instructions; (2) Worker Adjustment Annual Program Report and 
Instructions; (3) Example of state Follow-up Questionnaire; (4^ 
Sample Request for Proposal for Follow-up Contractor; (5) How 
Select a Random Sample; (6) Selected Survey Research Reference^, (7) 
Procedures to Adjust for Nonresponse Bias; and (8) Follow-up 
Softwak-e. (NLA) 
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Preface 



Passage of the Job Training Partnership Act (JTPA) in 1982 provided an 
innovative approach to employment and training, designed to address an important social 
and economic issue: Can American society bring unemployed or hard-to-employ 
individuals into or back into the productive workforce through a partnership of industry 
and government aimed at job training? Unlike previous employment and training 
programs, JTPA is based on specific performance standards designed to assess the 
effectiveness of ongoing programs and to reward programs that are effectively achieving 
the goals of JTPA. 

The bottom-line goal of JTPA is long-term employability-participation in the 
workforce that goes beyond short-term placement in a job. To obtain feedback on the 
effectiveness of JTPA programs in promoting retention in the labor force, the U.S. 
Department of Labor (DOL) required States or SDAs to collect information on the 
postprogram outcomes of JTPA participants, beginning October 1, 1986. Since that time. 
States or SDAs have been collecting data about the employment status and earnings of 
JTPA terminees during the 13-week period after leaving the program. 

To help States and SDAs collect follow-up data, the U.S. Department of Labor, 
Employment and Training Administration (ETA), issued the Follow-up Technical 
Assistance Guide for Postprogram Data Collection (TAG) in June 1986. In Program 
Year (PY) 1988, some States began to include in their incentive award calculations the 
SDA's performance on one or more of four postprogram performance measures. 
Consequently, valid and accurate follow-up data became a critical concern to ensure that 
incentives are distributed fairly among SDAs. 

This Technical Assistance Guide for Follow-up and Validation (TAG) is an 
update of the earlier TAG. It is designed to help all those involved in the JTPA follow- 
up process obtain accurate, valid follow-up data that meet the federal reporting 
requirements, that provide an equitable basis for incentive awards, and that yield useful 
information for program management decisions at all levels. 

In addition to providing current information on JTPA follow-up requirements, 
the TAG presents follow-up and validation strategies gathered in a recent study 
conducted for the DOL by The American Institutes for Research (AIR), Palo Alto, 
California. Follow-up staff in 50 States plus the District of Columbia and Puerto Rico, 
and 151 SDAs were surveyed to determine how follow-ups are being conducted, what 
efforts are taken to ensure the accuracy of the data collected, and what strategies are 
helpful in obtaining valid follow-up data. A detailed description of this study, including 



methodology, results, and recommendations, appears in AIR's Final Report: Study of 
JTPA Postprogram Follow-up Validation. 

The TAG is not intended to be a complete set of instructions for designing and 
implementing a follow-up system. Rather, it is intended to help States, SDAs, SSAs, and 
independent follow-up contractors adapt existing procedures or adopt new strategies to 
improve the follow-up data collection and to enhance the accuracy, validity, and 
ultimately the usefulness of the information obtained. 
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Section 1 - Overview of Follow-up and Validation 

This overview includes an introduction to JTPA follow-up and validation and a 
description of the purposes and organization of this Technical Assistance Guide for 
JTPA Follow-up and Validation (TAG). 

1.1 Introduction 

Since October 1, 1986, States have been required to collect and report to the U.S. 
Department of Labor (DOL) selected postprogram data at the Service Delivery Area 
(SDA) level for all adult and adult welfare terminees served under Title II-A, and at the 
Statewide level for terminees served under Title III. Beginning in Program Year (PY) 
1989, States are required to collect similar postprogram data on a Substate Area (SSA) 
basis for terminees of the Economic Dislocation and Worker Adjustment Assistance Act 
(EDWAA) program. 

The postprogram data collection requirements are based upon three performance 
measures that describe the labor market experiences of former JTPA participants over 
the prescribed 13-week follow-up period. These measures include the employment rate 
during the 13th full calendar week after program termination; the average gross weekly 
earnings of employed terminees during the 13th full calendar week after program 
termination; and, for all respondents, the average number of weeks employed during the 
13-week follow-up period. 

Beginning in PY 1988, these performance measures were used to award monetary 
incentives to SDAs that exceeded their standards. To assure that incentives are awarded 
equitably and that the adjustment models are adequate, it is essential that the follow-up 
data be accurate and valid and that the procedures used to collect these data be 
consistent and thorough. Without assurance of accuracy, validity, and uniform data 
collection procedures across SDAs/SSAs* in each State, incentive funds may not be 
awarded fairly, and follow-up data may not confidently be used in program management 
and evaluation. 

Implementing an effective follow-up system that meets these criteria requires the 
commitment and motivation of every individual involved in the follow-up process-from 
the JTPA terminees who provide the information, to the local SDA/SSA or contractor 



*In subsequent sections of this TAG, SDA is gcneraUy used to refer to SSAs, as weU as SDAs. 
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staff who help to locate the participants, to the follow-up interviewers who collect the 
information, to the JTPA officials at all levels who report and use the information. 
These individuals must clearly understand the purposes of follow-up, appreciate the 
potential usefulness of the data, and be motivated to do what it takes to obtain complete, 
accurate, valid information from as many terminees as possible. Ongoing communication 
among these individuals is essential also to ensure the timely transmission of information, 
to produce maximum response rates, and to provide adequate monitoring and quality 
control that will assure accurate, valid 'results. 

Purposes of the TAG 

This TAG is a supplement to the Instructions for the JTPA Annual Status Report 
(JASR) and the Worker Adjustment Program Annual Status Report (WAPR). An 
update of the original Follow-up TAG for Postprogram Data Collection (Version 3.0, 
June 1986), it is designed to help States, SDAs, SSAs, and independent follow-up 
contractors to: 

• understand and apply the DOL follow-up requirements for data 
collection and reporting 

• implement effective strategies for obtaining high response rates and 
valid follow-up data 

• obtain accurate, valid follow-up data that provide the basis for the 
fair and equitable award of incentives 

• use the information effectively for Federal, State, and local program 
planning and evaluation 

Because States and SDAs have been developing and implementing JTPA follow- 
up since PY 1986, they have had an opportunity to learn what strategies have or have 
not been effective in obtaining valid follow-up data within their particular setting. This 
TAG will provide States and SDAs an opportunity to assess what has and has not 
worked in conducting follow-up, as well as an opportunity to consider adopting strategies 
that have been used effectively elsewhere as part of an ongoing effort to evaluate and 
improve JTPA program components. 
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Organization of the TAG 

The TAG is organized into the following four sections: 
Section 1 - Overview of Follow-up and Validation 
Section 2 - Collecting Follow-up Data 
Section 3 - Reporting Follow-up Results 

Section 4 - Assuring the Accuracy and Validity of Follow-up Data (Validation) 

Each section is divided into subsections according to critical tasks involved in the 
follow-up and validation process. Requirements are presented at the beginning of each 
subsection, where applicable, followed by reconunended strategies for meeting the 
requirements and enhancing accuracy and validity. Illustrative tables, figures, and 
samples appear as exhibits within the text. Additional resource materials appear as 
appendices. 

Once it has been read from beginning to end, the TAG is intended to be a 
reference guide. Certain sections of it will be more useful to those individuals who are 
responsible for the corresponding follow-up tasks. Use the section and subsection 
headings to determine which sections are most appropriate for your needs. For example, 

• Everyone involved in the follow-up process should be familiar with 
the Overview in Section 1, 

• Staff responsible for collecting the follow-up data-particularly 
interviewers and their supervisors-should focus on Section 2, 
Collecting Follow-up Data. 

• Those involved in data analysis and reporting as well as staff 
involved in management and program evaluation should refer to 
Section 3, Reporting Follow-up Results. 

• All persons involved in collecting, processing, reporting, and supervising 
follow-up activities should refer to Section 4 on Validation. 
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1.2 Federal Requirements for Collecting and Reporting Follow-up Data 

Performance standards for PY 1988 and beyond include three postprogram 
measures, each of which reflects a different aspect of the labor market experiences of 
former participants during the 13 weeks following termination from a program. States 
and SDAs (or SSAs in the case of EDWAA) are required to report these measures 
separately for Title U-A adults, Title II-A adult welfare recipients and dislocated 
workers. No follow-up information is required for Title II-A youth. 

Required Follow-up Data 

The follow-up measures include: 

1. Employment rate at follow-up 

This measure is defined as the percentage of total respondents 
(those with completed interviews) who were employed full-time or 
part-time during the 13th full calendar week after termination: 

Number of respondents employed at follow-up 
1 1 X 100 

Total number of respondents 

2. Average weekly earnings (of employed respondents) at follow-up - Title II-A 

For Title II-A this measure is defined as the sum total of gross 
(before tax, health and other deductions) weekly earnings of 
employed respondents during the 13th follow-up week, divided by 
the total number of respondents employed during the 13th full 
calendar week after termination. Average weekly earnings is based 
upon only those respondents employed during the 13th follow-up 
week: 

Total gross weekly earnings of 
respondents employed during the 13th week 



Total number of respondents employed during 
the 13th week 
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Averagt hourly wage (of employed respondents) at follow-up - EDWAA 

For EDWAA, this measure is defined as the sum of gross (before 
tax, health and other deductions) hourly wage rates of employed 
respondents during the 13th follow-up week, divided by the total 
number of respondents employed during the 13th full calendar week 
after termination. Average hourly wage is based upon only those 
respondents employed during the 13th follow-up week: 

Sum of gross hourly wage rates of 
respondents employed during the 13th week 



Total number of respondents employed during 
the 13th week 

3. Average number of weeks worked in the follow-up period 

This measure is defined as the sum of the number of weeks worked 
during the full 13-week period by all respondents, divided by the 
total number of respondents. This results in an average number of 
weeks worked in the follow-up period for all respondents regardless 
of whether they worked at all during the 13-week postprogram 
period: 

Total number of weeks worked in the 
13-week follow-up period for all respondents 



Total number of respondents 

In addition to these measures, the overall response rate and the sample size must 
be reported on the JASR/WAPR for adults, adult welfare recipients, and dislocated 
workers. These items are being used to develop i statistically sound, nationally 
representative database to support valid postprogram standards and adjustment 
procedures. 

Instructions for completing the PY 1989 JASR and WAPR appear respectively in 
Appendices A and B. (Follow-up information appears in Section III of both reports.) 
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Required Response Rates 



When information from follow-up interviews is used for making critical decisions, 
the validity of the information must be assessed. One criterion for assessment is the 
response rate. The response rate is the percen. that actually responded completely to 
the questionnaire out of the total number of possible respondents. (An interview is 
complete only if the respondent answers all of the required questions.) The lower the 
response rate, the lower the degree of confidence in the follow-up data. Having 
confidence that SDAs are performing well or poorly is critical when States reward or 
sanction SDAs. By establishing minimum required response rates, the DOL ensures that 
data are adequate for statistical modeling and that the States can be confident of 
meeting the DOL guidelines for postprogram standards. 

Minimum response rates of 70 percent must be obtained for each of the following 
six groups: 

• Adults Employed at Termination 

• Adults Not Employed at Termination 

• Adult Welfare Recipients Employed at Termination 

• Adult Welfare Recipients Not Employed at Termination 

• Dislocated Workers Employed at Termination 

• Dislocated Workers Not Employed at Termination 

These requirements apply to each SDA/SSA for Title II-A and EDWAA, and to each 
State still reporting follow-up data for Title III. 

Nonresponse bias is a problem that develops when ceitain groups are 
systematically over- or under-represented in the completed follow-up interviews. 
Differences in response rates between respondents who were employed at the time of 
termination and those who were not employed may bias the data: People who are 
employed at termination are more likely to be working at 13 weeks; consequently, they 
may be easier to locate, have higher earnings, and work more weeks in the 13-week 
period than people who were not employed. If people employed at termination are 
over-represented in the completed follow-up interviews, it is more likely that the follow- 
up data will be biased toward higher employment rates and higher earnings. 

For this reason, maximizing response rates for all groups of terminees is 
important to all States and SDAs involved in follow-up. Special attention should be 
given to successfully contacting and interviewing terminees who were not employed at 

1.2 Federal Requirements - 7 



14 



termination. Where the response rates between these two groups are substantially 
different, adjustments for nonresponse bias must be made. Complete instructions for 
making these adjustments are in Section 3.1, Calculating Follow-up Statistics. 

Additional, Optional Follow-up Information 

The follow-up information described above are the minimum data that must be 
collected and reported to DOL to support the development and implementation of 
postprogram performance standards. In order to increase the value of the follow-up 
information for program planning, management, and evaluation. States and SDAs should 
carefully consider collecting additional information beyond the required data. 

In considering the collection of additional information, keep in mind that most of 
the time and expense required to collect follow-up information stems from locating and 
contacting terminees, not from conducting the interviews. Given the fixed cost of simply 
reaching the respondents successfully, it can be very efficient and cost effective to gather 
additional information. 

Additional information might include any or all of the following: 

• Uutudes and opinions regarding JTPA training: A critical area to 
consider is collecting information regarding respondents' perceptions of the 
quality and value of the training received, their need for additional 
services, and suggestions for program changes. Asking such questions is 
useful for monitoring and evaluating programs and service providers, 
identifying problems, providing helpful feedback to service providers, and 
establishing immediate rapport with follow-up respondents. 

Examples: How and why did you enroll in JTPA? 

Was JTPA useful in getting or keeping a job? 

How would you rate the program? 

What was the best thing about the program? worst? 

What one thing would you do to improve the program? 

• Employment history since lea\ing JTPA: Collecting information on 
terminee employment status and activity throughout the 13-week 
follow-up period can provide information that will also be useful in 
evaluating programs and making critical management decisions. 

1.2 Federal Requirements - 8 
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Movements into and out of the labor force can be tracked, as can 
shifts between jobs and between employment and additional 
education and training. Detailed information can be collected about 
particular jobs, including hours, wages, benefits, occupation, industry, 
name and location of the firm, and opportunities for advancement. 
Information on EDWAA/Title HI . ^nnincc industry and occupation 
is particularly important since the goal is to re-employ workers in 
new industries and occupations. - interviewers can gather this 
information through a series of structured questions, using the 
reference week, a calendar, and time cues to jog the respondent's 
memory. 

Examples: If currently employed: What is your job title? hours worked? 
fringe benefits? medical benefits? 
If not employed: Are you currently looking for a job? If yes, how 
are you looking? If no, what is the reason you are not 
looking? 

• Receipt of other benefits. In keeping with JTPA's goal to reduce 
welfare dependency, collecting information on the respondent's 
welfare status (e.g., type of welfare, benefit amount) may be 
considered. Consider also collecting data on the respondent's 
receipt of unemployment insurance and disability benefits. 

Example: Are you currently receiving any form of public assistance? 
If yes, how much are you receiving? from whom? 

Appendix C presents an example of one State's questionnaire. 



Additional, Optional Follow-up Data Collection 

In addition to collecting supplementary information from respondents. States and 
SDAs may consider collecting other data that can be useful for broader management and 
evaluation needs. Specifically, data for additional groups of termlnees can provide 
information on a wider range of JTPA participants than Title II-A aduhs and dislocated 
workers. Data on additional periods besides the 13 weeks after termination can provide 
valuable information on longer-term retention and earnings. 
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• Groups of terminees. In addition to collecting infonnation for the 
required groups (Title II-A adults and adult welfare recipients and 
EDWAA/Title III dislocated workers), States and SDAs can collect 
data for other groups: Title II-A youth, Title II-A 3%/8% special 
projects. Title II-B summer youth, and Title IV-C veterans. If large 
enough numbers are involved in the program, you can also select a 
sample of a group, selected by service provider and/or program. 

• Reporting Periods. In addition to collecting information for the 
prescribed 13-week follow-up period. States and SDAs may consider 
collecting data at additional intervals~for example, 26 or 52 weeks 
after termination. Some States conduct 26-week and/or 52-week 
follow-ups to gather information on longer-term retention and wage 
growth. The nature of programs for dislocated workers (EDWAA/ 
Title III) especially suggests that substantial change in earnings, 
occupation, and industry might only occur over a longer period of 
time than 13 weeks. Some SDAs also conduct interim 30- or 60-day 
follow-up to maintain contact with all terminees prior to the 
required follow-up or to gather information needed for local vendor 
monitoring. 

• Data Sources. In addition to the use of participant interviews. 
States may consider using employer surveys and/or administrative 
record data to supplement their postprogram data collection efforts. 
These additional sources, of course, cannot be used for JASR/ 
WAPR reporting in lieu of information gathered directly from JTPA 
terminees. 

Summary of Requirements and Options 

Exhibit 1.1 summarizes the DOL follow-up requirements and options discussed 
above as well as additional requirements described in subsequent sections. 
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Exhibit 1.1 



JTPA Follow-up Data Collection: POL Requirements and Options 



Requirements 


Options 


WHO 

1. All II-A Adult Terminees* 

9 All TT-A AHnIt Wplfarp Terminees* 

3. All EDWAA Terminees* 

*or sample if planned number is more ihan 131 


1. n-A youth terminees 
? 3% 8% terminees 

3. n-B summer youth 

4. Selected target groups (out of school youth, 
offenders, etc.) 


WHEN 

Data collected following 13th full 

jiftpr tprminatinn and heforc 

18th full week after termination 

Data recorded on JASR/WAPR 
within 45 days after end of 
Program Year 


1. 26 weeks after termination 
2. 12 months after termination 


WHAT 

1. Employment rate at follow-up 

9 Averape weeklv earnings fll-A^ of 
employed respondents only 
(average hourly wage for EDWAA) 

3. Average number of weeks worked 
ai follow-up for all respondents 


1. Attitude/opinions 

2. Labor force status 

3. Employment information 

4. Dependency status 


1. Questionnaire by phone 
or in person 

2. Mail, if no telephone 


For 26 week or 12 month follow-up, 
mail or employer contact could also 
be used. 



1.2 Federal Requirements • 11 



18 



13 Using Follow-up Information for Program Evaluation and 

Management 



The purpose of collecting accurate JTPA follow-up data is much broader than 
meeting Federal reporting requirements. Follow-up systems should be designed as an 
integral part of JTPA program management at the local, State, and Federal levels. 
However, because JTPA postprogram standards and the requisite follow-up process are 
relatively new components of JTPA, States and SDAs focused during PY 1986-88 on 
designing and implementing follow-up data collection systems to meet the minimum 
reporting requirements. Although States and SDAs acknowledge the value of follow-up 
data, they sometimes lack the energy and resources required to use these data to their 
full potential. 

One of the most challenging aspects of postprogram follow-up is using the follow- 
up as a tool for reviewing and improving JTPA programs and performance standards at 
the Federal, State, and SDA levels. The relatively recent introduction of follow-up 
standards reflects JTPA's increasing emphasis on long-term employability. As such, 
making follow-up an integral part of JTPA program evaluation and management by 
providing accurate and timely follow-up information is critically important. 



Uses of the Follow-up Data 

Using follow-up information to evaluate and improve JTPA programs at the State 
and local levels, and as a source for developing national standards, has many implications 
for decisionmakers and managers at all levels. Follow-up data may present evidence of 
JTPA's success to government officials and lawmakers, representatives from business and 
industry including PIC members, and the public at large, and thereby can strengthen 
JTPA's credibility. In addition, officials at the Federal, State and local levels can consider 
follow-up data for key planning decisions related to overall service mix and client 
targeting. By linking demographic and program information with follow-up data, they 
can compare the effectiveness of various program activities from a longer-term 
perspective. 

At the State level, in addition to using follow-up data in the granting of incentive 
awards to SDAs, follow-up information can be used to develop and encourage statewide 
coordination of programs. Favorable follow-up results can demonstrate the credibility 
and effectiveness of JITA and thereby facilitate the establishment of statewide 
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collaborative efforts. States may also find follow-up data useful in determining where 
technical assistance is needed and in evaluating the effectiveness of State-level program 
initiative S"f or example, comparing the pre- and postprogram wages and employment 
status of EDWAA/Title III dislocated workers to determine how effective the JTPA 
program was. 

Follow-up data can play a critjcal role at the local level also. By assessing the 
employment status and earnings of participants 13 weeks after termination, SDAs can 
improve program monitoring and contract reviews. In particular, follow-up information 
can be used to assess individual service providers more comprehensively, compare the 
effectiveness of various program activities, and evaluate the quality of services provided 
to specific target groups. SDAs could integrate follow-up criteria into performance- 
based contracts to provide assurance of results that go beyond enrollments, completions, 
and simple placements. In addition, SDAs can use follow-up data to evaluate their 
internal client services. For example, if the data suggest poor retention after OJT 
placements, an SDA might examine the extent to which inadequately prepared clients 
are referred to OJT or whether the OJT employers are more interested in temporary 
training subsidies than in training permanent employees. By improving accountability for 
longer-term employability, SDAs can significantly improve their job training and 
placement services. 

At the local level also, follow-up information should be used by PICs to 
periodically review the bottom-line results of JTPA programs in relation to expected 
performance. Follow-up standards provide the PIC with a measure of results and the 
beginning of an answer to the important question: Are local JTPA programs increasing 
long-term employability and earnings of their participants? Follow-up information can 
also be used to evaluate the quality of the jobs and benefits provided by employers, 
particularly if additional information regarding benefits (e.g., medical and paid leave) is 
collected. 

In summary, follow-up data, linked with appropriate information such as selected 
demographic variables, training programs, and service providers, can be an invaluable 
tool to: 

• Monitor overall JTPA effectiveness in terms of long-term 
employability goals 

• Predict welfare reduction and other long-term impacts 
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• Evaluate and compare the effectiveness of different JTPA programs and 
service providers 



• Identify problems in programs and service providers that require 
further monitoring or corrective actions 

• Assess follow-up results in terms of labor market conditions 
(average wage, unemployment rate, population density, number of 
families below the poverty level) 

• Analyze follow-up results in terms of important demographic factors 
(sex, age, ethnicity, welfare status) 

• Compare current follow-up results with previous results to identify 
trends, identify problems, and recommend corrective actions 

• Compare follow-up results with the performance of similar SDAs or 
States 

• Identify and provide outreach to clients who require further services 

• Provide vital information on program effectiveness to PICs and 
State Job Training Councils responsible for planning and oversight 

• Provide the basis for technical assistance to States, SDAs, and 
follow-up contractors 



Importance of Communicating Follow-up Results 

Depending on the level of decisionmaking, the follow-up data can be used on an 
individual SDA-basis only or aggregated by States or the Federal Government to provide 
broader indicators of program effectiveness. In order for the data to be of any use, it is 
essential that JTPA staff at all levels be Informed about the follow-up results in a timely 
manner. For example, in order for SDA staff to monitor, evaluate, and improve local 
JTPA programming, they must have prompt access to follow-up information, whether the 
data are collected at the local or State level. Similarly, for management decisions, like 
establishing incentive policies, that are made at the State level, follow-up information 
must be made available to State decisionmakers. Finally, for management decisions, like 
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setting perfonnance standards, that are made at the Federal level, the follow-up 
information must be forwarded to and used by DOL. 

Involving All JTPA StafT in the Follow-up Process 

Involving JTPA staff at all levels in the initial planning and design of follow-up is 
highly recommended. In particular. States and SDAs should consider the benefits of 
collecting additional information, beyond the required follow-up data, that will help 
program evaluation, reporting, and decisionmaking at both levels. Involving everyone in 
the planning, implementation, and utilization of follow-ups is highly motivational as well: 
everyone is more likely to "buy into" and support follow-up as a valuable part of JTPA 
programming and evaluation, and will be more committed to producing accurate and 
valid follow-up information. Communicating the results of the follow-up to all involved 
is especially critical to reinforcing its importance and usefulness to JTPA staff at all 
levels. 



Limitations in the Usefulness of the Data 

Despite the value of using follow-up information, decisionmakers need to 
understand the limitations of even completely accurate information obtained on all 
terminees in an SDA. The meaningfulness of the data depends on size of each group 
being examined. For example, a small SDA that wants to evaluate differeiit programs in 
terms of their follow-up employment rates might not be able to make reliable 
conclusions based on small numbers of terminees from each program. A more detailed 
discussion of the limitations in interpreting follow-up data appears in Section 3.2, along 
with guidelines for using the data appropriately. 



1.3 Using Follow-Up Information - 16 



?2 



1.4 Organizing the Follow-up Interviewing 

Organizing the follow-up interviewing requires joint decisions by States and SDAs 
on two issues: (1) whether the interviewing should be centralized Statewide or 
decentralized (conducted by SDAs), and (2) whether the interviewing should be 
conducted in-house or by an outside follow-up contractor. These decisions have 
implications for both States and SDAs in terms of quality control, staffing, financial 
resources, and fair distribution of incentives across SDAs. 

Since the beginning of required follow-ups in October 1986, States and SDAs have 
adopted and sometimes changed their follow-up systems to best meet their individual 
management needs and their available resources. The majority of States have 
centralized follow-ups as indicated in Exhibit 1.2, which presents the types of follow-up 
systems in place during PY 1987 and 1988: 

Exhibit 1.2 

Number of States* and T^pes of Follow-up Systems 

PY 1987 PY 1988 

Centralized Follow-up 

State - In-house 12 H 

State - Outside contractor 24** 26** 

Decentralized Follow-up (SDAs) 16 15 



'includes District of Columbia and Puerto Rico 

"includes one State which L centralized except for one SDA 



Regardless of the model adopted. States, SDAs, and outside contractors, where 
involved, must continually communicate and cooperate in implementing, evaluating, and 
modifying the follow-up system to assure accurate and valid results. In evaluating your 
system, consider the effectiveness of your current follow-up in terms of the advantages 
and disadvantages discussed below. 
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Centralized vs. Decentralized Follow-up 

Beginning with PY 1990, all States will grant incentive awards to SDAs based to 
some degree on postprogram performance. The benefits of centralized follow-up are 
related closely to the equitable distribution of these awards. First, a centralized system, 
whether operated in-house or contracted out, has better monitoring and quality control. 
This stems from the administrative efficiency of operating a single, centralized effort and 
the resultant consistency of follow-up procedures implemented across all SDAs in the 
State. 

Second, by centralizing the effort. States can more easily ensure that the data 
have been collected and reported in an objective, consistent, and comparable way. 
When data are collected by persons directly involved in the program being studied (i.e., 
by SDA staff), they may be subject to criticism as being biased. In addition, uniformity 
of procedures enhances the utility of the information by allowing comparisons to be 
made across target groups, programs, and geographic areas. States can grant incentive 
awards based upon follow-up data, confident that the monies have been awarded 
equitably among SDAs across the State. Providing the same assurance for a 
decentralized process may require extensive State monitoring. 

Third, a centralized system can be more efficient and cost-effective, particularly 
where there is a statewide MIS capability. Centralization allows administrative 
economies of scale to be realized through lower fixed or negotiated costs. Centralization 
also diminishes the duplication of effort needed to monitor, validate, and provide 
technical assistance for follow-ups in each of the SDAs throughout the State. The need 
for monitoring and validation is discussed in greater detail in Section 4 of the TAG. 

The advantages of a decentralized system in which SDAs conduct follow-up, 
however, should not be overlooked. First, SDAs in control of their own follow-up may 
be able to produce higher response rates because of greater familiarity with their 
terminees and more thorough and timely access to information needed to track and 
locate terminees. Second, SDA staffs are better able to stay in touch with terminees, 
provide outreach services, and be aware of the longer-term outcomes of their programs. 
Third, SDAs that "own" the follow-up may be more motivated to use the data for local 
planning, management, and evaluation needs; can customize their follow-up to better 
meet these needs; and have immediate access to the data. Conversely, they often lack 
the resources and training to accomplish comprehensive follow-up. 
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The decision to centralize or decentralize the follow-up effort requires the close 
involvement of both State and SDA-level staffs in defining mutual roles and 
responsibilities and in cooperatively implementing the follow-up process. 

In-house vs. Contracted Follow-up 

Whether centralized or decentralized, follow-ups can either be conducted by 
in-house staff or contracted out to private or public survey research organizations. The 
advantages to keeping the follow-up in-house are that the time and effort involved in the 
procurement process are avoided and the additional costs involved in an outside contract 
are not incurred. Monitoring the follow-up process is easier, and the risk of hiring a 
poor contractor is eliminated. With in-house interviewing, however, existing staff may be 
burdened with the additional responsibilities of follow-up or extra staff must be found 
and trained to do the interviewing. The work load may fluctuate, requiring changing 
levels of staffing. Staff may be unable to work during evening and off-hours when most 
terminees can be reached. 

Hiring an experienced contractor has several adventages. A carefully selected 
contractor can implement a follow-up system efficiently, with minimal startup problems 
and minimal disruption to ongoing State or SDA operations. A qualified contractor 
should have the survey research facilities and capabilities to conduct quality, low-cost 
interviews and to produce high response rates. For example, contractor staff are 
prepared to do interiewing evenings and weekends, while State or SDA staff may be 
reluctant to work such schedules. A contractor is highly motivated to produce high 
response rates in order to retain the follow-up contract. Finally, a contractor can 
enhance the public perception of follow-up as objective and impartial, and thereby 
improve the credibility of the data since the contractor has no vested interest in the 
follow-up results. 

If you decide to hire a contractor, be sure to assess the organization's capabilities 
carefully. Look for these characteristics in a contractor: 

• previous experience conducting interviews with programs and 
participants similar to those in JTPA-preferably familiarity and 
previous experience with JTPA follow-up and with economically 
disadvantaged populations 

• track record in achieving high response rates in telephone surveys 
with similar clientele 
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• adequate telephone interviewing facilities, including multiple phone 
lines 

• an incoming, toll-free 800-line 

• a compatible MIS system ihat can receive, process, and transmit 
data to the State or SDA agency 

• adequate staff and staff training (including, if necessary, bilingual 
interviewers for terminees with limited English ability) 

• reasonable cost per completed interview 

In issuing a Request for Proposal (RFP) for prospective contractors, you should 
include the essential requirements for all anticipated work including any monitoring and 
verification efforts required, the frequency and format of reports, and any option for 
contract renewal. Also include specifications for minimum levels of performance, such 
as response rates, or penalties for missing deadlines and for errors or omissions. The 
RFP can be relatively brief and to the point; an example appears in Appendix D. 

General Guidelines to Promote Accurate and Valid Follow-up Data 

Whether your follow-up is centralized or decentralized, conducted in-house or by 
an outside contractor, it is important to motivate everyone involved in the follow-up 
process to produce accurate, valid follow-up data and maximum response rates. Each 
group involved in the follow-up (State, SDA, and/or contractor) depends on the other 
for various parts of the process: sampling, locating and contacting respondents, 
interviewing, reporting, and using the data. The following practices should promote 
accurate and valid follow-up data: 

• Establish clear and specific procedures for meeting follow-up 
requirements, and distribute these procedures to all involved in 
follow-up. 

• Assign specific roles and responsibilities to each person involved in 
the foUow-up-at the State, SDA, and/or ijontractor level. 
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De'/ise a system to provide current, accurate contact information to the JTPA 
agency or contractor conducting follow-up. 

Publicize, promote, and communicate the importance of follow-up 
throughout the JTPA system-to clients, service providers, 
eiTiployers, follow-up contractors, and SDA and State staff. 

Assess the follow-up system you have been using to determine what 
has worked well and what has not; modify your system appropriately 
to improve results. 

Consider using incentives and/or sanctions, making sure that the incentives 
do not themselves introduce bias or temptation to misrepresent results. 
States and SDAs have effectively used incentives such as these: 

- Minimum response rates required to receive full State 
incentives for postprogram measures and/or contractor 
payments 

- Contractor payments based on number of completed 
interviews 

- Interviewers rewarded for completed interviews 

Rapidly communicate the follow-up results to all involved in the 
process to encourage use of the data for planning and evaluation 
purposes and to reinforce future follow-up efforts. 
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Section 2 - Collecting Follow-up Data 



This section presents the requirements and recommendations for collecting follow- 
up data, including sampling (selecting terminees for follow-up); timing (defining the 
follow-up period, scheduling interviews and reporting); locating and contacting 
participants; conducting interviews; training and motivating interviewers; and setting up a 
management information system (MIS) for follow-up. 

2.1 Sampling 

To meet Federal requirements, follow-up information must be collected and 
reported by SDAs/SSAs for three groups: Title II-A adult terminees. Title II-A adult 
welfare terminees, and EDWAA terminees. States must collect and report follow-up 
information for EDWAA terminees in statewide, regional, or industrywide projects and 
for EDWAA terminees in National Reserve funded projects. 

If the number of terminees in each group is large enough in a single program 
year, a State or SDA may collect data on a sample of terminees rather than on all 
terminees eligible for follow-up. If done correctly, sampling can provide aggregate, 
accurate estimates of postprogram performance, without the cost of contacting all 
terminees in a group. 

Note that sampled individuals must NOT be selected until alter they have 
terminated their JTPA programs-and NOT upon or during their enrollment. 



Requirements 

When an SDA or State wishes to interview a sample rather than all terminees, it 
must meet these requirements: 

1. For each SDA, separate samples must be drawn for Title II-A adult 
terminees and adult welfare terminees. A sample of dislocated 
worker (EDWAA) terminees is required for each SSA. Separate 
statewide samples must be drawn for two groups of EDWAA 
terminees: (1) terminees in statewide, regional, or industrywide 
projects funded under Section 302(c)(1)(B), and (2) terminees in 
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projects funded under Secretary's National Reserve Grants, funded 
under Section 302(a)(2). 

2. No participant in the follow-up group may be arbitrarily excluded 
from the sample. Therefore, it is critical that all terminee records 
be promptly entered in the database used for sampling. 

3. The process of selecting terminees for each sample must be random 

and must conform to generally accepted statistical practice. 

4. Each sample must be at least as large as either the minimum 
sample sizes or sampling percentages listed in Exhibit 2.1. As you 

can see, sampling cannot be used if fewer than 138 participants in a 
group are expected to terminate over the 12-monih period. In such 
cases, attempts must be made to contact all terminees in the group. 

5. The sampling percentage must be the same for all terminees for 
each group and for each periodic sample from the follow-up cohort 

(participants who terminated during the first three quarters of the 
program year and the last quarter of the previous program year). 

These sampling requirements provide States with the assurance that observed 
postprogram differences among SDAs are, in fact, real differences. This is particularly 
important in the case of Title II-A ^^rograms where SDAs receive incentives or sanctions 
based on postprogram performance measures. The requirements help to ensure: 

• Lack of bias - Random selection from the entire follow-up group 
with uniform sampling rates ensures unbiased estimates of 
performance that are unaffected by seasonal fluctuations or 
concentrations of some types of terminees. 

• High precision of the performance estimates in each SDA. The 
minimum sample sizes ensure that the estimated follow-up 
performance measures across SDAs are precise. There is only a 5% 
chance that an SDA with a follow-up employment rate five 
percentage points below its standard would have met its standard if 
all terminees had been included in the sample. 



2.1 Sampling - 24 

?.9 



Exhibit 2.1 



Minimum Sample Sizes for Follow-up 



Annual Number of Terminees 
in Population Eligible for 
Follow-up 


Minimum Sample 
Size 


Sampling 
Percentace 


1 - 137 


All 


100 


138 - 149 


137 


94 


150 - 159 


143 


92 


160 - 169 


149 


89 


170 - 179 


154 


87 


180 - 189 


159 


85 


190 - 199 


164 


84 


200 - 224 


175 


82 


225 - 249 


185 


78 


250 - 274 


194 


74 


275 - 299 


202 


71 


300 - 349 


217 


67 


350 - 399 


229 


62 


400 - 449 


240 


57 


450 - 499 


250 


53 


500 - 599 


265 


50 


600 - 749 


282 


44 


750 - 999 


302 


38 


1,000 - 1,499 


325 


30 


1,500 - 1,999 


338 


22 


2,000 - 2,999 


352 


17 


3,000 - 4,999 


364 


12 


5,000 or more 


383 


7.3 



2.1 Sampling - 25 



30 



• Consistency - Uniform sampling procedures improve the 
consistency of follow-up data across SDAs. 

Sampling Decisions 

The Federal requirements only establish minimum sample sizes or sampling 
percentages. Although surveying additional terminees may increase costs somewhat, 
States and SDAs should consider selecting samples larger than the minimum for the 
following reasons: 

• Surveying all terminees may be easier and more cost efTective than 
sampling for small States and SDAs, and even for larger States and 
SDAs that want to analyze differences among subgroups in the 
sample. For example, in an SDA with 150 terminees, the minimum 
sample size is 143. Following up all 150 terminees is easier than 
using a sampling strategy to eliminate seven people. Similarly, a 
large State might find it more meaningful and more cost effective to 
survey all terminees than to administer a weekly, stratified sample. 

• Program managers may want to compare and evaluate the follow- 
up performance of particular programs, service providers, and 
population subgroups (e.g., age, race, gender). These more detailed 
analyses require larger samples for each SDA than the required 
minimum sample sizes, which are designed to provide accurate, 
aggregate estimates of the performance of SDAs, SSAs, or States 
(for statewide, regional, and industrywide projects or Secretary's 
National Reserve Grants under EDWAA). 

• Oversampling can provide protection against fluctuations in the 
number of terminees during the year and elimination of terminees 
from the sample due to death, institutionalization (e.g., in hospitals. 
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prisons, nursing homes), or severe incapacitation that makes them 
unable to be interviewed for the entire follow-up period. 

• Surveying all terminees simplifies the procedures used to analyze the 
follow-up data since a weighting procedure must be used to 
aggregate SDA results into statewide totals unless each SDA uses 
the same sampling percentage. 

Sampling Procedures 

Preparing for sampling. The following steps must be taken prior to selecting the 
sample: 

• Define a regular time period for sampling: it can be weekly, 
biweekly, or monthly (recommended for most SDAs). 

• Make sure all terminations for the period are entered in the MIS by 
the time you plan to select the sample-preferably less than 9 weeks 
after the beginning of the period. Otherwise, you have broken a 
cardinal rule of random sampling, which requires that all terminees 
must have an equal chance of being selected for the sample. If the 
terminee record is not in the database, it has been arbitrarily 
excluded. 

• Develop lists of all adult terminees in the period. The lists must 
identify welfare status and ideally would contain primary and backup 
contact (at least two) information for each terminee. Generally, the 
SDA would prepare this list, though the State could do it through a 
centralized or networked MIS. The list should be prepared no later 
than 10 weeks after the beginning of the follow-up period. 

• Send the lists of terminees, ideally in machine readable form, to the 
group responsible for sampling (State, SDA, or contractor as 
appropriate) no later than 10 weeks after the beginning of the 
termination period. 
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Determining the minimum sample size or sampling percentage. The minimum 
sample size and sampling percentage were both designed to meet the same statistical 
criterion and differ only because of the use of ranges and rounding. States or SDAs may 
choose to use either method. For ease of explanation, "minimum sample size" is used 
below. 

The first step in sampling is to determine the required minimum sample size by 
locating the planned annual number of terminees for each group in the left-hand column 
of Exhibit 2.1. For each group of terminees eligible for follow-up, a corresponding 
minimum sample size is needed to estimate SDA-level follow-up performance for that 
group. The required minimum sample size appears in the middle column. The right- 
hand column expresses the minimum sample size a' a percent of the number of planned 
terminees. Either the minimum sample size or the sampling percentage can be used to 
determine a sample size that will meet the DOL requirement. For example, suppose an 
SDA estimates it will serve 550 adult II-A terminees in the Program Year. Exhibit 2.1 
shows that a minimum sample size of 265 or a sampling percentage of 50% is required 
for 550 terminees. The SDA must therefore select at least 50% of adult II-A terminees 
for each sampling p'^.iod throughout the year. 

Once a sampling percentage is determined, it must be applied throughout the 
whole year. Changing the sampling percentage during the year to correct for inaccurate 
estimates can bias the results of the follow-up. If the original number of planned 
terminees is too high, you cannot adjust the sampling proportion to meet the minimum 
sampling requirement. Thus, using a somewhat larger than minimum sampling 
percentage is recommended to assure meeting the sample size requirement. 

Sampling to reflect planned number of terminees accurately. Terminees must be 
sampled periodically (monthly or weekly) and consistently so that the total sample 
accurately reflects the flow of terminees from programs. The percentage of the size of 
the periodic sample to the size of the group of terminees ehgible for follow-up each 
period should equal the percentage of ♦he total sample size to the size of the total group 
eligible for follow-up. For example, if an SDA selects a sample size of 50% of all 
expected adult terminees for the year, it should draw a 50% sample each sampling 
period. In a period that had 150 terminees, the sample would be approximately 75. In a 
period with only 80 terminees, the sample would be about 40. The samples from all 
periods combined will equal the total sample size for the year, or 50% of all terminees. 

The entire samphng scheme depends on the accuracy of SDA plans because 
sampling proportions are based upon planned, or expected, numbers of terminees. 
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Exhibit 2.1 shows that the lower the number of terminees, the greater the percentage of 
terminees that should be sampled. Therefore, if actual terminations run behind plan, too 
few terminees will be included in the sample and the required minimum sample size may 
not be met. Conversely, underestimated planning figures can lead to oversampling and 
to higher than necessary sampling costs. 

The following procedures are provided to help States or SDAs obtain adequate 
and unbiased samples that reflect follow-up outcomes throughout the 12-month period. 
(These procedures were adapted from the State of Indiana's Sampling System, revised in 
April 1989.) 

1. To determine the total number of terminees expected to be served 
by an SDA, review the performance standards plan submitted by the 
SDA. Note the number of planned terminations for the upcoming 
program year. 

2. To verify these planning figures, review data for the past two years 
to analyze trends. First, compare the planning figures with the total 
actual number of terminees in each gvoup for each of the past two 
years. If the planning figures look re;ilistic compared to the past 
years' data, they can be used as the basis for determining the 
appropriate sampling percentage. Then, compare the sample sizes 
to the total terminations for the past two years to make sure that 
the sampling percentages used in the past were adequate and are 
appropriate for the upcoming year. 

3. Once you determine the "realistic" number of terminations for the 
upcoming year, use Exhibit 2.1 to establish the appropriate sampling 
percentage, as described earlier. 

4. Determine the number to include in each sample by multiplying the 
number of terminations falling within the appropriate range of dates 
by the sampling percentage. Round the result to the next whole 
number to provide a small, automatic cushion against 
undersampling. 

Using random sampling. The process for selecting terminees for inclusion in the 
sample must be random. That is, all terminees in each group must have an equal chance 
of being selected. No participant may be arbitrarily or systematically excluded from the 
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sample: individuals employed at termination and those not employed at termination must 
all have an equal chance of being selected. Any method consistent with accepted 
statistical procedures m?v be used for selecting the random sample. Several examples of 
random sampling methods appear in Appendix E (How to Select a Random Sample). 

Backup sampling. Individuals who have died or are incapacitated may be 
excluded from the sample and from the calculation of response rates. This includes 
persons who are institutionalized (e.g., in hospitals, prisons, nursing homes) or severely 
incapacitated and unable to be interviewed for the entire follow-up period. If 
eliminating such individuals drops the sample below the minimum size, you need to 
replace them in order to obtain the minimum sample size. Any excluded individual 
should be replaced by another termihee randomly selected from the same group 
(e.g., II-A adult) and month of termination as the excluded terminee. If, as previously 
recommended, States and SDAs initially draw random samples greater than the 
minimum size required, backup sampling is not necessary. 

Selecting the Title II-A M'elfare sample. The Title II-A adult welfare sample will 
be selected in two parts. First, some adult welfare terminees will be selected as part of 
the overall Title II-A adult sample. Then, you must randomly draw an additional 
supplemental welfare sample from those welfare recipients that were eligible for 
follow-up, but were not already selected in the adult sample. 

To determine the size of the supplemental welfare sample, determine the 
minimum sample size for the adult welfare recipients. Subtract from this number those 
welfare recipients already included in the total Title II-A adult sample. (The ratio of 
welfare adults to all adults in the sample drawn for follow-up may differ from the ratio 
in the total terminee population for the year. This discrepancy is not important.) The 
result is the minimum size of the supplemental welfare sample. An example follows on 
the next page. 

In many SDAs, the number of adult welfare terminees in a given year will be so 
small that a 100% sample would be necessary. In such cases SDAs would draw the 
overall adult terminee sample, picking up some portion of the welfare terminee 
population in the process, and then include all remaining adult welfare recipients in the 
welfare follow-up sample. 
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An SDA expects to terminate 1,600 adults in Title II-A during Program Year 
1989. Of these, 640 will be welfare recipients. Welfare recipients will 
represent 40% [(640/1,600) x 100] of the adult terminees. 

The required minimum sample sizes, taken from Exhibit 2.1 are: 
Minimum adult sample size = 338 
Minimum welfare sample size = 282 

If randomly drawn, approximately 40% of the adult sample could be expected 
to be welfare recipients. Forty percent of the adult sample of 338 equals 135. 
So the adult welfare supplemental sample would be 147, because: 

282 adult welfare sample size 
- recipients already in the a dult sample 
147 supplemental welfare sample 

The supplemental welfare sample of 147 individuals must be selected from the 
505 welfare recipients not previously selected for the sample (640 - 135 = 505). 
This represents a sampling percentage of 29% (147 divided by 505). 
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Requirements 

Three requirements are involved in timing the follow-up: the duration of the 
follow-up period; the time span, or window during which interviewing may occur; and the 
particular groups, or cohorts of terminees who arc followed up and reported each 
quarter. 

1. The postprogram, or follow-up period for all terminees is the 13 wggkg 
following termination from a JTPA program. For a given terminee, the follow-up period 
begins on the first Sunday following the termination date and concludes on the 14th 
Saturday following the date of termination. The last full calendar week of the follow-up 
period, from the 13th Sunday to the 14th Saturday, is designated as the reference week. 
The data reported in the JASR/WAPR should reflect the employment rate and average 
weekly earnings during the reference week, and average number of weeks worked during 
the entire 13-week follow-up period. (The data reported on the WAPR should report 
the average hourly wage, rather than the weekly earnings.) 

2. The time span, or window for data collection is limited to a 4-week period 
from the 14th through the 17th calendar week following termination. Postprogram data 
collection may extend beyond this 4-week window through the 18th week only when the 
individual has been located but not interviewed before the end of the 17th week or when 
the individual is temporarily absent during the regular 4-week period for data collection. 

3. Follow-up data must be collected and reported to DOL for individuals who 
terminated from the program in the first three quarters of the current program year, as 
well as those who terminated from the program in the last quarter of the previous 
program year. (This group is called the follow-up cohort.) This is because the follow- 
up interviews lag the termination dates by about 13 weeks. The schedule of quarterly 
cohorts for whom follow-up interviews are required in PY 1989 and PY 1990 appears in 
Exhibit 2.2. States and SDAs can schedule and conduct inurviews on a monthly or 
weekly basis, using a program year calendar to identify appropriate reference weeks and 
interviewing windows. The follow-up data must be entered on the JASR/WAPR and 
reported to DOL no later than 45 days following the end of the program year. 
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Exhibit 2.2 



Annual Required Follow-up Cohorts by Termination Quarter 



PY 1988 


PY 1989 


PY 1990 


Q4 


Ql 


Q2 


Q3 


Q4 


Ql 


Q2 


Q3 


Q4 



PY 1989 PY 1990 

Follow-up Cohorts Follow-up Cohorts 



Timing Decisions 

The follow-up interview schedule controls the timing of the entire data collection 
process. Exhibit 2.3 summarizes key activities in the follow-up process and recommends 
a schedule for conducting them in order to meet data collection and reporting 
requirements. 

Consider, in addition, these scheduling suggestions for conducting key follow-up 
activities: 

• Provide lists of terminees with accompanying contact information to 
interviewers no later than 10 or 11 weeks after termination. This 
allows time to prepare and mail notification letters, if used, and to 
schedule interviews. If the lists are provided to interviewers once a 
month, contact information will need to be generated as early as six 
weeks after termination. This process should be coordinated with 
the MIS data entry process. 

• Because the follow-up workload varies with the number of 
terminees, flexible stafling may be needed to accommodate peak 
periods of terminations. 

2.2 Timing - 34 



38 



When using mail surveys to contact hard-to-reach terminees, all 
attempts at telephone interviews should be made during the 14th 
week to allow enough time to mail questionnaires to terminees so 
that complete responses mailed by the end of the 18th week can be 
included in the data reported on the JASR/WAPR. 

The data collection window leads to a very tight reporting schedule. 
Consider a termination that occurs on the last day of the third 
quarter. The reference week ends on the first Saturday after the 
program year, followed by four weeks to complete the interview. 
Thus, the interview period ends approximately 31 days after the 
close of the program year, leaving only 14 days to enter, edit, and 
report the data to DOL. 
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Exhibit 2.3 



Summaiy of Key Activities 



TIME PERIOD 


TASK 


Prior to and during JTPA 
participation/termination 


• Inform participants of follow-up, and obtain 
their agreement to participate. 

• Collect and update primary and back-up contact 
information for all participants. 

• Promptly enter termination data into the MIS. 


Weeks 10-11 
Post-Termination 


• Prepare list of terminees eligible for follow-up. 

• Select sample(s). 

• Prepare notification letters. 


Week 13 
Post-Termination 


• Prepare questionnaires, as applicable. 

• Mail pre-notification letters. (Send earlier if using 
reply cards.) 


Weeks 14-17 
Post-Termination 


• Contact and interview all terminees included in 
sample. 

• Make callbacks, as needed. 

• Code and enter questionnaire data, as required. 


Weeks 18-19 
Post-Termination 


• Complete remaining interviews and data entry for 
terminees who mail back questionnaires by the end 
01 me loin weeK. 


End of Program Year 


• Prepare aggregated reports for required groups. 

• Adjust for non-response, as required. 


45 days after 

End of Program Year 


• Submit JASR/WAPRs to DOL. 
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2.3 Locating and Contacting Participants 



States, SDAs, and follow-up contractors have found that JTPA tcrminees rarely 
refuse to provide follow-up data, even relatively sensitive information like their income 
or welfare status. The rate of refusals ranges from zero to less than five percent. 
Therefore, the main challenges in conducting an effective follow-up are in locating and 
contacting terminees. Achieving a 70% response rate requires a comprehensive and 
efficient system to do this. This section presents the Federal requirements for locating 
and contacting terminees and recommended strategies for obtaining maximum response 
rates. Appendix F, Selected Survey Research References, cites several standard survey 
research texts that cover these issues in more detail. 



Requirements 

Only one general requirement must be met in locating and contacting participants 
for follow-up: The techniques used to contact terminees must not introduce bias. The 
same effort should be made to contact all types of terminees regardless of their 
termination status or demographic characteristics. In particular, similar effort must be 
made to locate individuals who were not employed at termination (and are typically 
harder to contact) and those who were employed at termination. 



Obtaining Adequate Contact Information 

The key to effective follow-up data collection is the quality and accessibility of the 
contact information obtained from participants while they are still in the program. The 
following techniques have been used to obtain this information effectively: 

• Upon enrollment, obtain the name, address, and telephone number 
of the participant and at least two backup contacts. These contacts 
should be relatives or friends who do not live with the participant 
but would likely know how to find the participant later on. "In case 
of emergency" contacts are not h^^^A if they are family members at 
the same residence, who may be as difficult to locate as the 
participant. Parents and other older relatives not living with the 
participant are ideal because they are less mobile. 
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• Require service providers to provide primary and multiple baclcup 
contact information on each terminee as part of their performance 
contracts. (This increased the response rate in one SDA by 5-10%.) 

• Periodically during training and upon termination, verify and 
update the contact information. 

- Give a prepaid postcard to participants to return with 
changed contact information. 

• Require participants to provide updated contact information 
on time sheets required for needs-based payments, where 
appropriate, 

- Feed the updated information into the MIS in a timely way. 

- Maintain contact with participants who are in the 90-day 
period of inactive status. Terminate them at the end of this 
period, and enter the data into the MIS promptly. 

• Establish linkages and develop cooperative relationships with other 
agencies and staff that might have contact with participants (SDA 
counselors, local employers, JTPA service providers, social services, 
correction agencies, community service organizations, homeless 
shelters). Encourage interviewers to use these resources after a few 
unsuccessful attempts to contact participants. A thank-you note 
might be sent to individuals who have helped locate terminees. 

• Communicate <*ontact information in a timely manner and make it 
easily accessible to follow-up interviewers-preferably through a MIS 
network. 

Preparing Participants for Follow-up and Motivating Them to Participate 

Terminees will be more willing to provide follow-up information if they know 
about follow-up in advance and understand its importance and legitimacy. The following 
steps are recommended: 
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upon enrollment and termination, inform participants of the 
importance of follow-up and their responsibility to provide follow- 
up information, if requested to do so. 



- Obtain the individual's signed agreement to participate in 
.follow-up and to provide contact information, at the time of 
enrollment. A sample Follow-up Agreement appears in 
Exhibit 2.4. 

- Require counselors or service providers to have an exit 
interview with participants to identify any need for additional 
services, to remind them about follow up, and io obtain 
updated contact information. Record the dale and the name 
of the exit interviewer in the MIS for use in establishing 
rapport in the follow-up interview. 

- Obtain the participant's permission to request information 
from employers. This will enable interviewers to call 
employers for help in locating employed terminees and in 
verifying follow-up data provided. An example of a 
statement used to obtain the participant's permission to 
contact employers is included in the Follow-up Agreement in 
Exhibit 2.4. (See Section 4.2 for more information on 
employer verification.) 
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Exhibit 2.4 



Sample Follow-up Agreement 

Participant Name SS# 

Program Operator 

You are being enrolled in a training program funded under the Job Training Partnership 
Act (JTPA). The program is administered by [SDA] and operated by [Service Provider]. 
Its goal is to hel' ♦ you obtain the education or training that will lead to your future 
employment. 

In order to measure how well we are meeting this goal, we need your help. After you 
leave this program, you will be contacted by [Follow-up Agency/Contractor] to see how 
you are doing and to ask you a few questions. By answering these questions, you will 
provide us with information that will help us to serve you and others better in the future. 

One of [Follow-up Agency/Contractor] interviewers will call you 14 weeks after you 
leave the program. It will take only a few minutes, and all the information you provide 
will be kept strictly confldential. Your participation is very important to the future 
success of this program. We appreciate your help. 

List at least two people who do not live in your household and will always know vour 
whereabouts. 



Name Address Phone 



Name Address Phone 



I voluntarily agree to provide information in the follow-up interview. I understand that 
my participation is voluntary and that the information I provide will be kept strictly 
confidential. 

I also hereby give permission to my past/present employer to release information to the 
[Follow-up Agency/Contractor] regarding my employment and earnings. 



Signature of Participant Date 



Signature of Program Date 
Representative 
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• About one week before the expected interview date, mail notification letters 
or post cards to follow-up participants: 



- Identify the purpose of the interview, its voluntary and 
confidential nature, and the specific information that will be 
requested (employment and earnings). The letter might 
include a calendar to highlight the reference weeks. 

- Remind them of previous references to follow-up, if 
appropriate--for example a Follow-up Agreement. 

- Provide a bilingual translation of the letter or post card for 
non-English speaking participants. If possible, these 
individuals should be identified in advance on the MIS. 
Translations of letters should be printed on the reverse side 
of the Enghsh letter, and both sides of the letter should 
indicate that the other version is on the reverse side. 

- Include methods to verify the accuracy of available contact 
information and to encourage participant to call in. For 
example, ask participants to provide their current phone 
number on a business-reply post-card or to call in on an 800- 
line or collect. Or subscribe to the U.S. Postal Sendee's 
Address Correction Service, which will notify you of any new 
address and its effective date, and charge only for successful 
tracking attempts (approximately $.30). 

A sample notification letter appears in Exhibit 2.5. Use or adapt it to meet your 
follow-up procedures. 



2.3 Locating and Contacting Participants - 41 



Exhibit 2.5 



Sample Notification Letter 
[SDA/CONTRACTOR LETTCRHEAD] 
[DATE] 

[PARTICIPANTS NAME] 
[PARTICIPANT'S ADDRESS] 

Dear [Mr., Ms.]: 

The [SDA/CONTRACTOR NAME] is interested in learning more about you and other 
persons who have recently participated in a Job Training Partnership Act (JTPA) 
program. While you were in the [NAME OF PROGRAM/VENDOR] several months 
ago, you may recall being told that you might be asked to participate in a follow-up 
survey after you left the program. 

Someone from our office will be calling you in a few days to ask you some questions 
about your employment experiences during the past several months. Information you 
provide will be used in evaluating the programs and will be held in strictest confidence. 
Your participation is voluntary and your answers will not affect your eligibility for 
AFDC, food stamps, unemployment insurance, or similar programs. 

Questions will include: 

1. Did you do any work for pay during the week beginning [START/END 
DATES]? 

2. If you worked during that week, how many hours did you work that 
week, how much did you get paid per hour, and how much extra, if 
any, did you earn in that week from tips, overtime, bonuses, 
commissions, or any work you did on the side before deductions. 

3. How many weeks did you work at all for pay during the entire 13- 
week period from [DATE] through [DATE]? 

A calendar is included with this letter to help you answer the questions. 

We will be calling you at [PARTICIPANTS PHONE NUMBER], the telephone number 
that is in our records. If you have a new telephone number, please write or call [NAME 
OF INTERVIEWER] at [PHONE NUMBER] so that we can get in touch with you. 

Those of us here at [SDA/CONTRACTOR] look forward to talking with you soon. 
What we learn from you will help us to improve the programs we provide. 
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Exhibit 2.5 (Continued) 



1989 



JANUARY 1989 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 
15 16 17 18 19 20 21 
22 23 24 25 26 27 28 
29 30 31 



FEBRUARY 1989 
S M T W T |- S 
12 3 4 
5 6 7 8 9 10 11 

12 13 14 15 16 17 18 

19 20 21 22 23 2; 25 

26 27 28 



MARCH 1989 
S W, T W T F S 
12 3 4 
5 6 7 8 9 10 11 
12 13 14 15 16 17 18 
19 20 21 22 23 24 25 
26 27 28 29 30 31 
30 



APRIL 1989 
S M T W T F S 
1 

2 3 4 5 6 7 8 
9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 



MAY 1989 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



SEPTEMBER 1989 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 
10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 



JUNE 1989 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 



OCTOBER 1989 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 31 



JULY 1989 
S M T W T F S 
1 

2 3 4 5 6 7 8 
9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 31 

NOVEMBER 1989 
S M T W T F S 
12 3 4 
5 6 7 8 9 10 11 

12 13 14 15 16 17 18 

19 20 21 22 23 24 25 

26 27 28 29 30 



AUGUST 1989 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



DECEMBER 1989 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 
10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 
31 



1990 



JANUARY 1990 
S M T W T F S 
1 2 3 4 5 6 
7 8 9 10 11 12 13 
14 15 16 17 18 19 20 
21 22 23 24 25 26 27 
28 29 30 31 



FEBRUARY 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 

11 12 13 14 15 16 17 

18 19 20 21 22 23 24 

25 26 27 28 



MARCH 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 31 



APRIL 1990 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 



MAY 1990 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



JUNE 1990 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 

10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 



JULY 1990 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 31 



AUGUST 1990 
S M T W T F S 
12 3 4 
5 6 7 8 9 10 11 
12 13 14 15 16 17 18 
19 20 21 22 23 24 25 
26 27 28 29 30 31 



SEPTEMBER 
S M T W T 



'.990 
F S 
1 

2 3 4 5 6 7 8 

9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 



OCTOBER 1990 
S M T W T F S 
1 2 3 4 5 6 
7 8 9 10 11 12 13 

14 15 16 17 18 19 20 

21 22 23 24 25 26 27 

28 29 30 31 



NOVEMBER 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 



DECEMBER 1990 
S M T W T F S 
1 

2 3 4 5 6 7 8 

9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 31 
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Contacting Participants for Follow-up 



Obtaining maximum response rates requires systematic and common sense 
procedures for contacting participants. After conducting follow-ups for several years, 
States and SDAs have found the following strategies effective: 

• Time your calls to reach participants successAiUy. Analyze your 
own contact records to determine the best times to call. In general: 

- Avoid calling during standard business hours when 
employed terminces are at work. Interviewers have found 
the best times to call are early evening hours (5 - 8 Monday 
through Friday), Saturday 10 - 2, Sunday evenings, holidays. 
Bad weather days are also good times to call. 

- Vary times for repeated calls if a participant is hard to reach. 

- If you reach an answering machine or someone other than the participant, 
leave a message for the participant to call back. 

In States or SDAs where calling during off hours is difficult for staff, 
consider offering compensatory time or monetary incentives to staff, 
rotating after-hours assignments among staff, or hiring special 
follow-up staff who can work flexible schedules. 

• Once you have reached participants, interview them immediately 
whenever possible. If completing the interview is not convenient at 
that time, arrange to call again at a more convenient time or for the 
person to call back easily: 

- Establish a toll-free 800-number if possible (it's easier and 
less intimidating to participants than calling collect). Forty 
percent of one State's response rate comes from callbacks on 
an 800-number, and several States have reported 
improvements in response rates of more than 20% from 
800-number callbacks. 
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- Whoever answers the phone must be able to conduct the 
interview or immediately transfer the call to someone who 
can. Assign a distinctive, nctitious name to the interviewer 
(that is neither male nor female, like Chris Jasser) so that 
return follow-up calls can be recognized and referred 
immediately to appropriate staff. 

• Establish systematic procedures for reach* terminees that include: 

- As many telephone attempts as necessaiy to reach the 
terminee: often six or more attempts are needed to reach 
participants. 

- a prescribed order for attempting contacts such as the following: 

1. Home phone 

2. Backup contacts 

3. SDA program staff 

4. Service provider (vendor) that trained participant 

5. Directory assistance 

6. Other directories such as local telephone books or cross- 
referenced/reverse directories that are published by private firms. 
Calls to other r ^.mbers listed under the same name or at the same 
address may provide information useful in locating a terminee. 

7. Employer or worksite of job-placed terminees 

Employers may provide useful information for locating terminees. 
By contacting the employer, it may be possible to speak with the 
participant to arrange an interview, leave a number for the 
participant to call, or get information on how to locate the 
participant. When calling employers, interviewers should be careful 
not to release confidential information about the participant, not to 
disrupt the participant's work day, and not to irritate an employer 
who could be the source of future placements. If possible, 
interviewers should establish a contact person at the employer, and 
call no more than once or twice in a week. Note that you cannot 
use any information about employment status or earnings obtained 
from employers for JASR/WAPR reporting purposes In Ueu of 
information obtained directly from JTPA terminees. 
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• a required record, or log, of all attempted contacts. The log 

should include the date, time, and method of each attempted 
contact; the name or ID number of the staff member making 
contact; and the outcome of each contact. It should be 
available for follow-up supervisors at all levels to use for 
verification and monitoring purposes. See Exhibit 2.6 for a 
sample Contact Log Form. 

• a follow-up mailing to hard-to-reach termlnees who have an 
active address on record. Mailing should be a last resort 
only, as it has not been a very effective initial follow-up 
strategy. (See Section 2.4 for more information on mailing 
questionnaires.) 

• Consider assigning the most hard-to-reach cases to a single 

interviewer or to a small number of especially qualified interviewers 
during third or fourth follow-up week. Although not practical for 
small SDAs, this strategy can improve the efficiency of a large 
follow-up system. 
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Exhibit 2.6 



Sample Contact Log Form 



COMPLETED J _ 
SDA: 
MAME: 
ADDRESS: 

PHONE NO: 
ALT. NOa 

WORK NO: 
NEW NO; K 



WEEK mt 

QUEST. »t 
SOCIAL SECURITY NO: 

TERM DATE: mtti/dd/yy 

ACTIVITY: 

TEF^M TYPE: i 



VENDOR: 
EMPLOYER: 
DATE: /. 



WA&E: »9i.'3'5 



TITLE: 

IIA= 
IIAU» 
111 = 
YOUTH= 
YOUTH WELF= 



DATES INTERVIEWED: / /. 

/ / 



CALL RECORD 



CALL 


DATE 


TIME AM«0/PM-l CALL CODE DISPOSITION 


CONTACT INrORMATlCf'4 




1 


/ 


/ 








*• 


/ 


/ 




















3 












4 


/ 


J 








5 


...^ 


J 








6 





J 








7 





J 








8 


1 


/ 


X 








f 


/ 


• 











J 


t 







14 15-16/17-18/19-20 21-24 2S 
CODES; 

l>CAlltd horrid 

1-CalUd kln/fri*nd 

2»Calltd Directory Assist. 

4«Calltd *tripl oy^rf 

S*C«lUd S^rvicr D*liv<fry Arr« 

bsCalUd training faciltiy 

7-Kart;cipant >;all«rd in 



NOTE: PUNCH OfiLY LAST -INE 



2& 27 
DISPOSITION CODES: 

0>No answer /busy signal (IP> < 

l«Particlpant not avallabU. R*sid*nc* Confir'«i«d IP* 

2«Int#rvi#w Partially Coi^pUtifd (IP) 

3«Phont # Disconn#ct»d/Chang#d/Not in S*rvio*/Nwt L.ived 
4«Participant no longer at i/Providifd no info:<iation 
5«Provid#d n#w afldrMs/phone i <1PJ 
6»Institutionali2*d/unabU to int#rvi*w (F) 
7«P*tusal on scr^*ning/t*niiinat#d during inttrr/.rw r 
e>Unabl# to contact (F) 
>Coi«pUt#d Int»rvi#w (F) 

23 Locating and Contacting Participants - 47 



ERIC 



51 



2.4 Conducting Follow-up Interviews 



Requirements 

To fulfill the follow-up reporting requirements, all of the following information 
inust be collected from respondents:' (1) whether the respondent was employed during 
the 13th week after termination (the reference week); (2) if employed, respondent's 
hours worked, hourly wage, and additional earned income in the 13th week; and (3) the 
total number of weeks the respondent worked during the 13-week period ending with the 
reference week. The following requirements for conducting follow-up interviews must be 
met: / 

• Interview questions developed by DOL must be used to determine 
the follow-up information reported on the JASR and WAPR. 

Participants must be told that responding is voluntary and that 
information they provide will be kept confidential. The required 
DOL questions appear in Exhibit 2.7. 

• Attitudinal questions may be asked before the required DOL 
questions. However, the required questions must be asked before 
any other questions about employment or earnings. 

• The required information must be gathered directly from the 
participant. Although relatives, friends, and employers may provide 
contact information, the questions must be asked directly of the 
participant. 

• The information should be obtained by telephone or in person. 
Mail questionnaires may be used in those cases where an Individual 
cannot be reached by telephone or in person. Telephone interviews 
should be the primary method for collecting follow-up data. In- 
person interviews may be appropriate when respondent? drop into 
JTPA offices, in rural areas without telephones, and for specific 
hard-to-reach terminees such as the homeless. Mail surveys should 
be a last resort only. More information regarding mail 
questionnaires appears later in this section. 
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Minimum Postprogram Data Collection Questions 

A. I want to ask you about the week starting on Sunday, , and ending on Saturday, , 

which was (last week/two/three/four weeks ago). 

1. Did you do any work for pay during that week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour in that week? 

Dollars per hour 

4. How much extra, if any, did you earn in that week from tips, overtime, bonuses, commissions, or 
any work you did on the side, before deductions? 

DoUars 

B. Now I want to ask you about the entire 13 weeks from Sunday, , to Saturday, . 



5. Including the week we just talked about, how many weeks did you work at all for pay during the 
13-week period? 

Weeks [Go to end] 

ALTERNATIVE QUESTIONS 
C If answered "NO" to Question 1: 

Now I want to ask you about the entire 13 weeks from Sunday, , to Saturday, , 

6. Did you do any work for pay during that 13-weck period? 

Yes [Go to 7] 

No [Go to end] 

7. How many weeks did you do any work at all for pay during that 13-wcek period? 

Weeks 
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Interviewing Participants to Obtain Complete and Accurate Information 

Establishing rapport. Once contacted, participants usually agree to provide 
follow-up information. However, they sometimes do not recognize "JTPA" or are 
reluctant to provide personal information such a' wages. Therefore it is important to 
establish rapport, trust, and cooperation immediately. Interviewers should introduce 
themselves and be friendly and informal. Tliey should briefly explain the importance of 
follow-up, its use for program improvement, and its voluntary and confidential nature. 
To convey familiarity with participants and concern about their JTPA experience, 
interviewers should: 

• Identify the participant's specific training program and service 
provider. (This information should be available through the MIS.) 

• If a notification letter was sent and to establish your credibility 
immediately, ask the participant if the letter was received. 

• Begin interviewing as soon as possible. The most efficient and 
effective way to convince respondents that you are not trying to sell 
them anything is to ask the interview questions immediately. 

• Ask participants for their opinion about their JTPA training before 
asking the required DOL questions-informally if these questions are 
not part of the actual questionnaire. 

A sample telephone introduction appears in Exhibit 2.8 at the end of this section. 

Clarifying questions and probing. Clarifying questions or probing is necessary 
when a respondent does not understand a question or when a response is incomplete, 
unclear, or probably inaccurate. Probing should be neutral: it should not lead the 
respondent to give a particular answer. Explanations, or clarifications, of the required 
questions appear in Exhibit 2.9, and examples of acceptable probes appear in 
Exhibit 2.10. 

Confronting language problems. If your State or SDA serves a substantial 
number of non-English speaking people, hiring or identifying a local source of bilingual 
interviewers is important. If language is an occasional problem, arrange for another 
person in the participant's home to translate, scheduling a callback if needed. However, 
make sure that the participant provides the information, not the interpreter. 

2.4 Conductbg Interviews • 51 

54 



Converting potential refusals. Once contacted, reluctant participants may be 
persuaded to complete the follow-up questions; similarly, reluctant friends or relatives 
may be encouraged to locate the participant. Reasons for possible refusals and 
suggestions for gaining cooperation appear in Exhibit 2.11. 

Using a mailed questionnaire. As mentioned earlier, use mail questionnaires only 
for terminees who cannot be reached by phone or in person. The cover letter should 
refer to the initial notification letter (if there was one), restating the purpose of the 
interview and explaining its confidential nature. It should also provide a return 
telephone number (preferably an 800-line) and hours when return calls can be made by 
respondents. Alternately, the letter might include a business reply postcard requesting a 
current name, address, and telephone number, and a convenient time for the interviewer 
to call. The mailed questionnaire should be sent in enough time to reach terminees by 
the end of the 17th week, so that complete responses mailed by the end of the 18th week 
can be included in the data on the JASR/WAPR. 

Experience of States and SDAs indicates that terminees are more likely to call 
back to respond than to return a mailed questionnaire, particularly a lengthy one. To 
increase the likelihood of a response, the mailed questionnaire should be brief and easy 
to read, fill out, and return. Include the required questions only, worded appropriately 
for a mailing, and a calendar for reference. Including optional questions in the mail 
questionnaire is not recomip'^nded. Be sure to include a self-addressed, stamped 
envelope. 

An example of a cover letter and mail questionnaire appears as Exhibit 2.12. 
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Sample Telephone Interview Introduction 

Hello, my name is [ ], and I am calling from the [Name of 

State, PIC, SDA, Contractor as appropriate]. As you know from the letter we sent you 
recently, we are conducting a brief survey of the [JTPA Training Program and Service 
Provider] in which you participated three or four months ago. 

I would like to ask you a few questions about what you have been doing since you 
left the program. All of your answers will be kept strictly confidential and are voluntary 
on your part. Your responses will help us to improve employment and training programs 
in the future. The questions are very short, so it will only take a few minutes. 

[Note to Interviewer: Do not sound hesitant or tentative. This may be the only time the 
participant will be reached. Push for completion, and at the very least, get answers to 
the required questions. If the participant cannot be interviewed immediately, schedule a 
later time to conduct the interview, or ask the participant to call back on the 800- 
number or collect.] 
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Exhibit 2.9 



Clarifications of Required Questions 



Required DOL Questions 


Clarifications 


Question Al: 

Did you do any worlc for pay 
during that week? 

Yes 

No 


The answer to this question should be "yes" if at 
any time during the 7 consecutive days of the 
reference week the respondent was: 

• a paid employee; 

• a member of the armed forces on active duty or 
a participant in a registered apprenticeship 
program; 

• self-employed in his/her own business, 
profession or farm; or 

• absent from work or the business due to illness, 
bad weather, vacation or personal reasons, but 
still on paid status. 


Question A3: 

How much did you get paid 
per hour on that week? 

$ hour 


The following items are included in hourly wages: 

• all wages, including pay for vacation and 
sick leave. 

• amounts which may have been deducted from 
the paycheck for taxes, health insurance, credit 
unions, retirement, child support, alimony, etc. 

• nouriy wages irom an jods neia aurmg reierence 
week. 

The following items are not included in hourly 
wages: 

• money received through pensions or transfer 
payments such as welfare, social security, or 
unemployment insurance. 


Question B5 and C7: 

Now I want to ask you about 
the entire 13 weeks starting 
from Sunday, 
to Saturday. 

[including the week we just 
talked about]. How many 
weeks did you work at all for 
payduring the 13-week period? 

Weeks 


This question asks the respondent to mentally 
calculate the total number of weeks worked during 
this 13-week period. Only whether the respondent 
has worked at any time in a given week is 
important, not the amount of time worked during 
the week. Again, individuals on paid sick leave 
or paid vacation should be counted as working. 

The interviewer may clarify the status of each of 
each of the 13 weeks with the help of a calendar, 
using the categories provided under question Al. 
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Acceptable Interview Probes 



Types of Probes 


Examples 


Repeat or rephrase 
a question 


^ere you paid for any work you did 
last week? 


Repeat or rephrase the 
respondent's answer 


You said you were paid $7.50 per hour, but I did 
not hear how many hours you worked last week. 


Make neutral statements 
requesting more information 


I'm not sure I understand. Could you explain further. 


Specify key events or dates 
to identify the reference period 
or suggest the respondent 
look at a calendar 


The 13th week was the week of Halloween. 


Probe or ask back-up 
questions to obtain earnings 
information-use available 
termination information to 
jog their memory. 


When you left the program, you were hired full-time 
at Acme Body Shop at $6.65 per hour. Were you 
making the same amount last week? Did you work 
40 hours last week? 


Clear up an apparent 
discrepancy error in a response 


I thought you said you were not working last week 
(the 13th week) but that you worked for the entire 
13 weeks since [date]. Did you really work all 13 
weeks or did you mean that you were working every 
week except last week? 
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Converting Refusals 



Reasons for Refusal 


Response 


Too busy/no time 


I only have a few questions, but would be happy to 
call you back at a more convenient time. When 
would be a good time for me to call back in the next 
day or two? 


Bad health 


I'm sorry to hear that. I would be happy to call back 
in a day or two. Would that be okay? 


Feels inadequate/confused 


The questions are not difficult. They concern how 
you feel about the job training program. Some of 
the people we have interviewed had the same concern 
you have, but once we got started they didn't have 
any trouble answering the questions. Would you like 
me to read a few questions to you and you can see 
what they're like? 


Questions too personal 


Your answers will be completely confidential. No 
one else will know how much you were paid. 
Protecting people's privacy is one of our main 
concerns. We add up the answers from lots of people 
to find out how well the program is working. 


Not interested/sees no benefit 


It's very important that we get the opinions of 
everyone who participated in the program. Your 
opinions and experiences with [program] will help 
us learn how useful the program was to you, so it 
will be useful for others. 


Objects to telephone surveys 


We have had a lot of experience and success with 
teleohone survevs like this one flf Darticioant is 
Still reluctant...] However, if it would, be convenient 
for you, I would be happy to mail a questionnaire to 
your home. 
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Exhibit 2.12 



Sample Mail Letter and Questionnaire 
[SDA/CONTRACTOR LETTERHEAD] 



[DATE] 

[PARTICIPANT'S NAME] 
[PARTICIPANT'S ADDRESS] 



Dear [Mr., Ms.]: 



Several months ago you participated in a job training program funded by the Job 
Training Partnership Act (JTPA). Our records indicate that you were enrolled in 
[NAME OF PROGRAM/VENDOR]. We are now conducting a follow-up survey to 
find out how successful the program was in helping you. You may recall being told 
about this survey while you were in the program. Your cooperation would be greatly 
appreciated. 

We have been unable to reach you by telephone, so we are sending you a brief 
questionnaire. Please take a few minutes to fill out jilJ of the items on the questionnaire, 
and return it to us in the enclosed stamped envelope. It is important that we receive 
your response before [Knd.date of 4 .wppk intgrvicw "window"], so answering quickly 
would be appreciated. If it would be more convenient, feel free to call [interviewer 
name] us with your answers at [800-number and/or other number] between [appropriate 
hours/days]. 

Your responses are voluntary and will be held in the strictest confidence. The 
answers you give will in no way affect your eligibility for this or other, similar programs. 

Thank you for your help and let us know if we can be of assistance to you in the 

future. 

Sincerely, 

Program Director /Survey Manager 

Enclosures 
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Exhibit 2.12 (continued) 

Sample Mail Questionnaire 

We would like to know about your work experiences during the 13-week period 

shown on the calendar below-starting on Sunday, , and ending on 

Saturday, . 

[Insert small calendar with appropriate weeks highlighted.] 

1. Did you do any work for pay during the 13th week, starting on Sunday, , 

and ending Saturday, ? 

Yes 

No [If NO, go to question 2.] 

la. How many hours did you work in that week? 
Hours worked 

lb. How much did you get paid per hour in that week? 
Dollars per hour 

Ic. How much extra, if any, did you earn in that week from tips, overtime, 
bonuses, commissions, or any work you did on the side, before deductions? 
Dollars 

2. How many weeks did you work at all for pay during the entire 13-week period, 
starting on Sunday, , and ending Saturday, ? 

Weeks worked 

Thank you very much. Please return this questionnaire in the self-addressed, stamped 
envelope as soon as possible. 
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2^ Training and Motivating Interviewers 



Training, supervising, and motivating interviewers are essential components of an 
effective follow-up system, whether the follow-up is conducted statewide or at the SDA 
level, in-house or by a contractor. Systematic training is important to familiarize staff 
with JTPA and the importance of follow-up, with the procedural requirements of follow 
up, and with strategies for obtaining accurate, valid data and high response rates. 



Training, Supervising, and Evaluating Interviewers 

Sample interviewer training materials appear in Exhibits 2.13, 2.14, and 2.15. 
These materials have been adapted from materials developed by Employment and 
Training Institute, ETI, a private follow-up contractor in Ringwood, New Jersey. Exhibit 
2.1? provides a suggested interviewer training outhne; details for most of the topics in 
the outline are provided in the TAG sections indicated on the outline. The Orientation 
Guide in Exhibit 2.14 provides information for Section I of the outline relating to the 
background and purpose of JTPA and follow-up data collection. An example of a form 
that can be used to evaluate interviewers appears in Exhibit 2.15. 

Providing interviewers with sufficient training and supervision can ensure that they 
understand the purposes and procedures involved in follow-up, can deal with hard-to- 
locate and reluctant participants, and will collect and record follow-up information 
completely and accurately. For example, new interviewers who have not previously 
talked with economically disadvantaged people may have difficulty remaining objective 
and not be tempted to offer personally to help. Helping interviewers develop techniques 
for remaining objective, while establishing rapport, would be valuaole. 

In addition to training and regularly monitoring the work of interviewers, the supervisor 
should meet regularly with interviewers to provide feedback, to discuss problems and 
effective ways to address them, and to share each other's effective strategies. 



Motivating Interviewers 

In addition to training, supervising, and providing feedback to interviewers, 
consider motivating interviewers in the following ways: 
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• Establish friendly, informal competition among staff by periodically 
reporting results. For example, post results for each interviewer in 
terms of completed interviews per day or most challenging locations 
reached. One State uses a wall map to plot the locations of faraway 
respondents interviewed. 

• Acknowledge or reward interviewers for achieving high response 
rates or using innovative strategies. For example, designate an 
interviewer of the month, and hold a monthly "ceremony" to 
acknowledge successful staff. Engage interview staff in the process 
of determining appf^priate incentives. 

• Pay interviewers or contractors for completed interviews. Consider 
paying bonuses for high response rates. (One State pays its follow- 
up contractor a bonus for each percentage point over a 70% 
response rate.) However, if incentives are used, monitoring 
interviewers is especially critical to ensure honest and thorough 
performance. 

• Improve staff accountability by integrating interviewer's 
performance on follow-ups into the performance appraisal process. 
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Training Outline for Interviewers 

I. Project Background and Purpose [Exhibit 2.14] 

A. The Job Training Partnership Act (JTPA) 

1. Funding 

2. Explanation of Private Industry Council (PIC) 

3. Explanation of Service Deliveiy Area (SDA) 

4. Discussion of participant eligibility for Title II-A and EDWAA 

5. Description of program activities (Classroom Training, OJT, Direct 
Placement) 

6. Description and role of community agencies (service providers, 
employment service, welfare) 

7. Organization of State/ SDA program staff 

B. Follow-up Data Collection 

1. Significance of results in terms of performance standards 
[TAG Sections 1.1 and 1.3] 

2. Use of results for evaluating and monitoring programs and service 
providers [TAG Section 1.3] 

3. Explanation of the 13 week follow-up period, reference week, and the 
"window" for data collection [TAG Section 2.2] 

4. State, SDA, or Contractor's experience with follow-ups [TAG Section 1.4] 

II. Procedures for Locating and Contacting Respondents [TAG Section 2.3] 

III. Review of Interview Questions and Procedures for Recording Responses 
[TAG Section 2.4] 

IV. Interviewing Principles and Procedures [TAG Section 2.4] 

A. Establishing rapport with respondents 

B. Responding to different types of respondents (typical, reluctant) 

C. Pacing the flow of the interview 

D. Probing to clarify questions and responses 

E. Reversing refusals 

V. Role Playing Exercise (mock interviewing) 
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nmriBni's omrmai ooisi 

TO JTPA AD PQ8TPI00IAM POXOH-OP 

As a Postprograa Pollow-Op Inttrvltwtr, you will b* in contact with ptopla who 
havt gona through tha JTPA (Job Training Partnarahlp Act) syataa. This guida 
will attaapt to acquaint you with thia aystaa. 



BACKQSOOID 

JTPA is a fadarally-fundad Jobs prograa gaarad priaarily to aarring tha aco- 
nomically disadvantagad. It has baan In aziatanca ainoa 1982. Pravioualy, tha 
fedaral Jobs prograa was callad CBTA (Coaprahansiva Eaployaant 4 Training Act). 

The Job Training Partnarahlp Act (JTPA), anactad in 1982, rapraasnta an antiraly 
naw concept in providing Job akill training for tha nation's unaaployad diaad- 
vantaged and dislocated workers. Funded in 1984 at $3.6 billion, this Act 
channels federal dollars through the atatea to looal pritata industry councils 
for training progress. Tbeae local councila auat be chairad by business 
persons, and local business leadera auat aake up a aajority of the council 
aeabershlp. 

The local private and public leaders serving on private induatry councils 
deteraine what eaployaent skills are needed in their ooaaunlties. They can then 
use federal funds to establish looal trslnlng prograaa that provide the 
disadvantaged uneaployed with the skilla to aatoh identified local eaployaent 
needs. The councils continually review prograa operations and evaluate actual 
results such as the percentage of traineea eaployed, oost per placeaent, and 
other perforaanca criteria. They then deteraine if new or different akill 
training prograas are needed. 

The Job Training Partnership Act la designed to insure that public funds are 
spent effectively to provide needed Job skills. . Seventy percent of all funds 
Bust be used for training prograaa that lead to private aactor eaployaent. 
Adainistrativa costs are liaited to a aaziaua of 15*. At leaat 40} of the funds 
aust be used to train disadvantaged and uneaployed youth— the nation's aost 
critical category of uneaployaenU (Minority youth uneaployaent ia now -jvar 
40$. Non-ainority youth uneoployaent is over 20% today.) 



THE TRAIVDIG PROGRAM 

Based on the decisions aade by the PIC, public and private training schools are 
chosen (through a bidding process) to train JTPA-eligible participants into Jobs 
(often entry level). The Act providea that a wide variety of training services 
aay be nade available to youth and adults with prograa funds. These services 
Include, but are not liaited to: 

* Job Factory or Job Search Assistance — A group of activitiea which 
provide labor aarket inforaation, prograaa for individuals and groups, 
Job clubs, and individualized Job search plans, . This assistance can be 
beneficial and inexpenaive (with very liaited supportive services 
provided) for clients with a aarketable Job akill. Job aearoh assistance 
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Exhibit 2.14 (continued) 



alao Bay b% a follov-up aotitity for olitota vbo bayt aueeaaafullx 
coapltttd a akilla tralnlot prograB. 

* Diract Placaiant ~ Tha prograa raorulta allfflbla clianta aod r%f%n thai 
to an axiating Job opaDlng. Vary llttla oounaallag or tralnltif la 
providad. 

* Pra-7ooational Trainlog — Gaoarally a ^aaarooa, atruoturad actlTlty 
vbleb providaa labor aarkat InforaatioD, introduotion to tba vorld-of- 
vork, inatructlon on oorract eoaplatiofi of raauaaa and aaploysant 
applicationa, and an avaranaaa of raquirad vorkplaoa babayior. 

a or Bl-Lingual Education ~ Spaciallaad inatruotion for non-Engliab 
and lialtad-Engliab apaaking clianta daaignad to aaka ttaaa aufficlantly 
fuoctional in Engllab to procaad vitb tbair iodlTldual aaployabllity 
plan* 

* CRT - Ranadial Education and Baaic Skllla Training Qanarally pro- 
grajBsad instruction to laprova raading, writing and ooaputational akilla 
of clianta who bava aducational dafielanclaa. (Tbia la oftan callad 
••Pre-Vocatlonal Training.") 

* High School Equivalancy (GED) Ganarally an actlTlty, oouplad with 
raaedial aducation, which la daaignad to proTlda an aqulTalancy cartifi- 
cate to individuala lacking a high acbool diplosa. 

* On-tha*Job Training (OJT) — A Job plaoaaant actlTlty which proTldaa for 
raiaburaaaent to tba amployar of an aaount aqual to aa auob aa 50$ of tha 
antry-leval aalary of an aaployaa, OJT ooapanaataa tba aaployar for 
axtraordinary auparviaion and Batarlal coata and raducad productlTlty 
during tha naw aaployaa*a training pariod. OJT prograaa ara gana^^ally 
viavad aa tha aoat affactlTa Job plaoaaant toola bacauaa tba aaployar la 
ablt to acraan and to aalact tba oaw aaployaaa and than train that, 
according to that aaployar* a apacific naada. 

* CRT - Occupational — Traditional ocoupation-'apaclfio training, which la 
providad by a public or privata training Tandor with dasonatratad 
training capability. SDAa aay buy daaa-alsa training or purcbaat 
indlTldual training alota, or Bay nagotiata with tba Tandor to cuatoaiza 
a apacial prograa to train for an Idantlflad daaand akill. 



SPECIAL TOOTB PROGRAMS 

Beyond tha aarvicaa daacribad ao far, tba local SDA Bay cbooaa to fund 
additional prograjna to aaat tha apacial aaployaant naada of youth in the 
labor aarkat araa. Tba Act daacribaa four auob prograBai any ona or all of 
which may ba included In tba Job training plan# 

* Pre-Eaployaant Skllla Training Prograa An in-*acbool prograa for youth, 
including 14 and 15 year olda, daaignad to provide up to 250 boura of 
inatruction and actlvitiaa in a variety of labor aarkati Job aearchi and 
baalc aklll areaa. Priority abould ba glTen to tboaa not aeeting 
eatibllabed levela of acadaaic aebieTeBent who plan to enter the full- 
tiae labor aarkat upon leaving acbool. 



2,5 Training and Motivating Interviewers • 63 



06 



Exhibit 2.14 (continued) 



Entry Eaployntnt Szptrltnot Program — A progru proTldlog lubsldliwl 
•iployaeDt opportunltlM In tbt publlo and prlTatt aactors for 
Indlvldutla vho ara anrollad In saooadary lattl tduoatlon, vltb priority 
to tboaa not planning to go on to poat^saooodary •duoatloni and vho matt 
otber rastrlctlona. 

Scbool- to-Work Tranaltlon Asslatanct Prograa i prograa dlractad at 
blgb school atniors planning to antar tha full*tlaa labor atrkat upon 
graduation. Transition sanricas aay inoluda Job saarch tasistanot and 
rsfsrral to training if nacasaaryi Including raftrral to adult training 
activitits and tha Job Corps. 



FUIDIIQ 

JTPA funding is al3ocattd by "subgroups" for tha purposs of ssrving diffarent 
segments of a community's eligible population. Subgroup funding sources are 
referred to as "titles". For examplet Title IXA funds programs which serve the 
economically disadvantaged. This category includes adulta» youth and veterans. 
Title IIA receives the majority of funding through JTPA. Title III monies fund 
programs for dislocated workers, according to a percentage of the tliglble 
population. 



THE PABTICIFAIT 

This section will, hopefully, give you an idea of the process a participant goes 
through in order to receive JTPA services. Although the JTPA process varies 
from state to state and SDA to SDA, the following Is a general overview. 

Many participants come into the JTPA system through looal Employment Services 
(Unemployment Offices, Jot Service); others through City aad County Welfare 
(AFDC^ WIN); some are recruited by individual training programs (Vendors), and 
many others through word-of-mouth referrals (friends or relatives who have gone 
through a JTPA program). 

In order for an individual to find out if he la eligible for a JTPA program, be 
must bring documentation of Income, age, address, family alse, etc Baaed on 
economic need (or other eligibility), the peraon la "certified" aa eligible. He 
usually is then tested academically (and, in some cttes, for aptitude). Each 
training program has its own entrance requirementa which must be met before 
referral. The individual then meets with a counselor to discuss his goalSi 
abilitiesi and test results. Based on that interview, a referral is made. The 
participant may be referred to classroom training, OJT, special programs, or a 
social service agency. Supportive services such aa transportation and/or child 
care allowance aay also be made available. 

Many JTPA participants have social, educational, economic and psychological 
barriers to overcome in order to complete a training program and/or obtain 
employment. For these reasons, many drop out of programs. 
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Exhibit 2.14 (continued) 



Tou will bt inttrvltvlog partldpanti vbo coapltttd trtlnlnf (rtftrr^d to u 
*po8ltltt"} and aont who did oot ooBplttt training (rafarrad to ai 'non-poai- 
tiva"^). Tha aaaa rafaranoa ia applioabla to OJT partioipanta. JTPA alao uaaa 
codaa to raflaot tba activity of a participant during hia anrollaant, Tou vill 
racaiva a list of UotlTltr Codaa* ao you vill knov if jou ara addra^alng 
aoaaona who baa gona througb claaarooa trainingi OJT| or apaoial prograaa. 

posTPioQUN raxov-op 

This ia tba fourth yaar tbat follow-up ia baing oonductad. Tba Dapartaant of 
Labor baa aaodatad that all JTPA prograaa conduct a follow-up of its partioi- 
panta (this can ba dona by Stata or SDA). 

In tha paati JTPA could only Judga ita affaotiTanaaa and parforaanca in taraa of 
positiva placanants (thosa placad in Joba upon taraination froa prograaa); thara 
has baan no way to assaas tha prograa'a long-tara raaulta. 

Follow-up on aacb participant will ba conductad at 13-vaak intarvala (aftar 
laavlng the prograa). Tba intarviaw will ravaal tha participant's work history 
(or lack of ona) ainca laaving tba prograa, and alao balp ua gain aoaa insight 
into tba participant's axpariancaa with tha JTPA systaa* 

Inforaation gainad froa this study will allow ua to dataraina, in part, waak- 
nassaa which aight axist within a particular JTPA ayataa, poor work hiatory 
aaong cartaln subgroups, and poor ratantion rataa raaulting froa placaaant 
through individual vandors. 



CORCLUSIOi 

Postprograa Follow-Up ia now baing conductad acroaa tba natioa Tba raaulta of 
this atudy will ba analyzed and validated by aganciaa on atata and national 
levels. 

Ab an interviewer, you ara tba aoat critical coaponent of the prograa. Tou 
vill be receiving tha responses froa partioipanta which will aaka thia atudy 
possible. 

The purpose of Postprograa Follow-Op is to aacartain the effectiveness of JTPA 
prograaa over the long tera. Hopefully, this study will produce inforaation 
which will be utilized to iaprova JTPA's overall ability to train a qualified 
and productive workforce* 
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Exhibit 2.15 



Interviewer Evaluation 



Interviewer Date 

Supervisor 

General Intemew Performance 

Veiy Good Msim Needs Work 

• Rapport with respondent (1(1 (1 

• Pace of interview (1(1 (1 
(ability to control questions) 

• Flow of interview (1(1 (1 
(smooth transition) 

• Accuracy in asking questions (1(1 (1 

• Verbal skills (1(1 (1 
(conversational quality, use 

of inflection) 

• Probing I 1 I 1 I 1 
(neutral, secures appropriate 

responses) 

• Good judgment (1(1 (1 
(responds well to challenges) 

• Accuracy in recording information I ] I 1 (1 



Commenis 

Specific comments on interviewer's performance 



Supervisor recommendations 
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2.6 Setting Up a Management Information System (MIS) for Follow-up 



Requirements 

There are no official requirements for Management Information System (MIS) 
procedures related to follow-up data- collection. However, certain data collection and 
information retrieval procedures are essential to an effective follow-up effort. The 
availability and accessibility of up-to-date addresses and telephone numbers of terminees 
and at least two backup contacts are the keys to locating most of the follow-up sample. 
A listing of client characteristics will enable interviewers to confirm that they have 
reached the correct respondent, will help them to establish rapport, and will idemify 
non-English speaking individuals who may need bilingual interviewers. Many 
respondents may know the type of training they received and their service provider but 
not the SDA name or "JTPA." Therefore, information on the participant's training 
program and service provider, as well as their employment status at termination, 
employer's name, and termination wage, will enable interviewers to establish credibility 
and jog the respondents' memory of their JTPA experiences. 




Making Adequate MIS Information Available 



Whether the MIS system is manual or automated, the following information about 
each terminee in the sample should be retrieved and made available to interviewers: 

• Name 

• SDA 

. JTPA Title (n-A Adult, II-A Adult Welfare, EDWAA) 

• Current telephone number and mailing address 

• Backup telephone number and address of at least one, but preferably two, 
people who can locate a respondent who cannot be reached (not 
"emergency numbers" of relatives who live with the terminee) 

• Age 

• Gender 

• Ethnicity 

• Native language and whether the terminee has limited English 
abilities to match the interviewer's language to that of the terminee 
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• Key dates including date of termination and start and end dates of 
the follow-up period and the reference week 

• Name, provider, location, and type of training received 

• Status at termination (placed or not placed) 

If placed at termination: 

• Placement employer's name, address, and telephone number 

• Wage at placement 

Increasing the Accuracy and Usefulness of the Follow-up Data 

To facilitate follow-up and to increase the accuracy and usefulness of the follow- 
up data, you need to establish procedures for providing adequate MIS information to 
interviewers, for editing and entering the follow-up data into the MIS, and for analyzing 
and reporting the data in a meaningful way. With only five postprogram reporting 
elements on the JASR/WAPR, Federal reports could probably be completed manually, 
particularly for small follow-up efforts. However, processing and meaningful analysis of 
follow-up data, especially of large data files, are best done by computer. Most States, 
SDAs, and contractors today have automated MIS systems that can be used to streamline 
follow-up operations. The following strategies have been used effectively: 

• Provide to interviewers a single computer-generated form for each 
terminee that includes participant data and a record of follow-up 
contacts. The questions can also be included. (An example of such 
a form appears in Exhibit 2.6 in the previous section of the TAG.) 

• Use a machine-readable form to simplify data entry. 

• Establish a Computer-Assisted Telephone Interview (CATI) system 
to eliminate the need for separate data entry and to ensure 
complete and accurate data collection and reporting. (See 
Appendix H for a description of the software to support JTPA 
follow-up.) 

• Build in MIS edits to check for internal consistency and 
reasonableness in the follow-up data. Use local labor market 
conditions and conmion sense to establish appropriate limits-for 
example, that an acceptable hourly wage ranges from $3.80 - $15.00, 
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and that hours worked during the 13th week range from 0-70. Use 
related data items to identify discrepancies-for example, a large 
difference between the wage at placement and follow-up weekly 
earnings. The DOL uses the following edit checks for consistency 
and reasonableness of SDA-level data when processing your 
JASR/WAPR data: 

- For Title II-A terminees, the Follow-up Weekly Earnings are compared to 
the Average Wage at Placement: 

Follow-up Weekly Earnings 
Average Wage at Placement 
The acceptable range is 30 - 55 hours. 

- The Employment Rate at Follow-up is compared to the Average Weeks 
Worked: 

Average Number of Weeks Worked x 100 

+ Follow-up Employment Rate 

13 

Tlie acceptable range is 0.8 to 1.2. 

Implement procedures for timely investigation of suspect data and 
error correction. 

Establish efficient programs for merging participant data with the 
follow-up data in order to produce reports that provide information 
useful for evaluation and decision-making. 
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Once the follow-up data have been collected, the results must be compiled and 
reported to DOL according to their specifications. Certain calculations must be made to 
comply with requirements and to ensure that the data reported accurately represent the 
employment status and wages of eligible respondents during the 13-week follow-up 
period. This section describes the requirements for calculating and reporting response 
rates and for completing the follow-up portions of the JASR and WAPR. It also 
describes guidelines for interpreting and reporting follow-up data for State and local 
evaluation and management purposes. 



3.1 Calculating Follow-up Statistics 

As explained in Section 1.2, minimum response rates of 70% must be obtained for 
each of six group of terminees surveyed. The value of the data is even greater if a 
higher response rate can be obtained. Response rates must be calculated separately for 
Title II-A adults and adult welfare recipients and for EDWAA dislocated workers who 
were employed at termination and those who were not employed at termination. 
Separate calculations are needed in order to determine whether either group is 
underrepresented among respondents. Follow-up statistics should be adjusted for any 
nonresponse bias introduced by differences in response rates. They must be adjusted if 
the difference in response rates between individuals who were employed at termination 
and those who were not is more than five percent (5%). 



Defining Response Rate for Follow-up Data Collection 

The response rate is defined as the number of terminees who provide complete 
follow-up information required by DOL, divided by the total number of terminees 
included in the follow-up sample. It is expressed as the following percentage, rounded to 
tenths (e.g., 73.6%): 

# terminees with complete follow-up interviews 
— X 100 

# terminees selected for follow-up 

The denominator, or base, of the response rate will be the number of terminees 
in the follow-up sample. Even if the State or SDA is following up all eligible terminees 
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or more than the minimum required sample size, the denominator must include every 
terminee thu was selected. 

An interview is considered complete only if the terminee has provided answers to 
all of the required follow-up questions. Responses such as "don't know" or "don't 
remember" to the required questions are not counted as acceptable answers. For 
example, an interview is not counted as comphtc if the terminee provides information 
about employment status and earnings during the 13th follow-up week but cannot 
provide the number of weeks worked during the follow-up period. This terminee would 
be excluded from the numerator, but would be included in the denominator in 
calculating the response rate. 

In calculating the response rate, persons institutionalized (e.g., in hospitals, 
prisons, nursing homes) or those who have died or are severely incapacitated (and 
therefore are unable to be interviewed for the entire follow-up period) should be 
excluded from the sample. I'he number of terminees who fall into these categories is 
generally quite small. All other terminees selected for the follow-up sample must be 
included in the denominator. 



Adjusting Follow-up Statistics for Nonresponse Bias 

As described in Section 1.2, nonresponse bias occurs when certain groups of 
terminees are over- or under-represented among the completed follow-up interviews. 
The follow-up results must accurately reflect all selected terminees, not merely those 
from whom responses could be obtained. Therefore, where the response rates between 
two groups of terminees (e.g., those who were employed at termination and those who 
were not) are substantially different, adjustments for nonresponse bias must be made. 

Adjustments are made by weighting the follow-up data collected according to the 
proportion of individuals employed at termination and the proportion not employed at 
termination. Adjusting for nonresponse bias is a straightforward process which can be 
done manually following the procedures presented in Appendix G. Appendix G 
includes: 

• a worksheet and instructions for calculating differences in response 
rates to determine whether nonresponse bias adjustments are 
required 
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• a worksheet and instructions for computing adjustments for 
nonresponse bias 

Alternately, the adjustments can be done by computer using the "Report" option 
on the software to support JTPA follow-up, which is described in Appendix H. An 
example that illustrates the manual adjustment procedure follows: 

Example of Nonresponse Bias Adjustment 

Suppose that: 

• the group eligible for follow-up includes 200 individuals employed at 
termination and 100 individuals not employed at termination. The 
SDA is following up on a 100% sample. 

• 180 of those employed at termination (90%) completed the 
interview, and 85% of these respondents (153 of them) were 
employed at follow-up. 

• 60 of those not employed at termination (60%) completed the 
interview, and 40% of these respondents (24 of them) were 
employed at follow-up. 

Then, using the procedures presented in Appendix G: 

• The correct (or adjusted) follow-up employment rate would be 70% 
not 73.8%: 

Correct (adjusted) InCQITgtt (P9t a<»iystgd) 

200 X 153 
180 

70% 

300 

170 + 40 
300 

In the correct (adjusted) result, each group contributes to the overall 
statistic in proportion to its representation in the total terminee population, 
not in the completed sample. 



100 X 24 



60 



73.8% - 



153 + 24 



180 + 60 
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Failing to adjust for nonresponse bias will proportionately magnity the 
performance of groups with higher response rates and discount the performance of 
groups with lower respo^ise rates. In other words, if those employed at termination have 
higher response rates (as they typically do), failing to make the adjustment produces a 
combined follow-up performance that weights the information more for respondents 
employed at termination than for the respondents not employed at termination. 
Although the adjustment does not correct for all potential nonresponse biases, it does 
diminish the bias. 

States or SDAs are required to adjust for nonresponse bias for all three 
postprogram measures: employment rate at follow-up, average weekly earnings, and 
average weeks worked. An adjustment must also be applied to each of the groups 
requiring follow-up data collection: Title II-A adults. Title II-A adult welfare recipients, 
and EDWAA terminees. 

Presently, the adjustment for nonresponse bias is required only when the 
difference in response rates between those terminees who were employed and those who 
were not employed at termination exceeds five percentage points. If the difference 
between the response rates is five percent (5%) or less, adjusting for nonresponse is 
optional since its effect on follow-up performance may be quite small. Nevertheless, a 
uniform procedure that makes the adjustment in all cases is recommended. 

Calculating the Required Follow-up Data 

Accurately calculating th^. required follow-up data is a critical prerequisite to 
reporting and utilizing valid follow-up information. Instructions for calculating and 
reporting the required follow-up measures appear in detail in the JASR and WAPR 
instructions (Appendices A and B). Instructions are summarized below, in terms of 
responses to the required follow-up questions, presented in Exhibit 2.7 (Q refers to the 
question number, and R, to the total number of respondents): 
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Employment rate at follow-up: Divide the number of respondems who 
answered '*yes" to Question 1 (employed at follow-up) by the total number of 
respondents (R). Then» multiply the result by 100 to get a percentage. 



Number of respondents employed at follow-up Sum [Ql] 
X 100 = X 100 

Total number of respondents R 

Average weekly earnings (of employed) at follow-up - (Title II-A/JASR): 

Divide the sum of the products of the answers to Question 2 (hours per week) 
and Question 3 (hourly wage) plus the sum of the answers to Question 4 
(other earned income) by the number of respondents who answered "yes" to 
Question 1 (employed at follow-up). 

Total gross weekly earnings of 
respondents employed during the 13th week Sum [(Q2 x Q3) + Q4] 



Total number of respondents employed during Sum [Ql] 

the 13th week 

Average hourly wage (of employed) at follow-up - (EDWAA/WAPR): Add the 
sum of the answers to Question 3 (hourly wage) to the sum of the answers to 
Question 4 (other earned income) divided by Question 2 (hours per week). 
Divide the result by the number of respondents that answered "yes" to 
Question 1 (employed at follow-up). 

Sum of gross hourly wage rates of 
respondents employed during the 13th week Sura [Q3 + Q4/Q2] 



Total number of respondents employed during Sum [Ql] 

the 13th week 

Note that the average hourly wage/weekly earnings of employed respondents 
at follow-up is based upon only those respondents employed during the 13th 
follow-up week. 
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• Average number of weeks worked in follow-up period: Divide the sum of the 

answers to Question 5 (weeks worked, employed at follow-up) plus the sum of 
the answers to Question 7 (weeks worked, not employed at follow-up) by the 
total number of respondents. 

Total number of weeks worked in the 
13-week follow-up period for all respondents Sum [Q5] + Sum [Q7] 



Total number of respondents R 
the 13th week 

Note that this calculation is based on all respondents, not just those 
employed. It results in an average number of weeks worked in the follow-up 
period for all respondents regardless of whether they worked at all during the 
i3-week postprogram period. 

• Sample sizes Report the total number of terminees selected for contact, 
including both respondents and nonrespondents. Only deceased, 
institutionalized (e.g., in hospitals, prisons, nursing homes), or severely 
incapacitated and unable to be interviewed for the entire follow-up period can 
be excluded from the sample. 

Note: All sums are to be based on answers from all respondents in the 
appropriate group (II-A Adult, II-A Adult Welfare, EDWAA). Exclude from all 
calculations any answers from nonrespondents (i.e., terminees who do not answer all the 
relevant required questions). 
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3.2 - Reporting Follow-up Results 



Reporting Results to the DOL 

Follow-up data collection and reporting for Title II-A adults and adult welfare 
recipients and for dislocated workers under EDWAA address the DOL's need for 
accurate, consistent data that will permit confidence in the measures and comparisons of 
program performance across States and SDAs within States. Once the follow-up data 
have been collected, they must be reported to the DOL on the JTPA Annual Status 
Report (JASR) for Titles II-A terminees, and the Worker Adjustment Program Annual 
Program Report (WAPR) for EDWAA terminees. Separate WAPRs must be submitted 
for (1) each SSA; (2) participants and terminees in statewide, regional, or industrywide 
projects funded under Section 302(c)(1)(B) of EDWAA; and (3) participants and 
terminees in projects funded under Secretary's National Reserve Grants (Section 302(a) 

(2). 

The following data must be reported in Section III of each report: 

• Employment Rate At Follow-up 

• Average Weekly Earnings of Employed At Follow-up (JASR) 
Average Hourly Wage of Employed at Follow-up (WAPR) 

. Average Number of Weeks Worked in Follow-up Period 

• Sample Size 

• Response Rate 

Data for the first three items (employment rate, earnings, weeks worked) must be 
adjusted for nonresponse bias, as described in the previous section of the TAG. The 
"Report" option on the software to support JTPA follow-up can be used to produce a file 
with the required information, in a format that can be printed for the required 
JASR/WAPR. (See Appendix H.) 

Exhibit 3,1 presents Section III. Follow-up Information from the cunent JASR. 
Complete copies of the PY 1989 JASR and WAPR, along with instructions for 
completing them, appear as Appendices A and B respectively. 
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Exhibit 3.1 



JASR Section III. Follow-up linformation 
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You must provide the DOL with completed follow-up information for the 
program year no later than 45 days after the end of the program year-that is, by August 
15, Because of the scheduled time period allowed for reporting, the data reported to the 
DOL by August 15 may not be completely edited. Therefore you should report updated 
data to the DOL as soon as possible after new data are entered and edited into the MIS. 
The DOL will include the updated information in the database used for devising the 
regression model worksheets and in its own annual reports. 



Reporting Follow-up Results for State and Local Management 

As discussed in Section 1.3, States and SDAs can use the follow-up data as a 
valuable management tool. In addition to reporting the required JASR/WAPR 
information regarding employment status and earnings 13 weeks after termination, 
reporting these results in relation to participant characteristics, programs, and service 
providers is very important. Where follow-up is centralized, it is especially important 
that States provide comprehensive follow-up data to the SDAs within the State in a 
timely manner. 
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Reporting formats. State and local JTPA administrators can determine how best 
to combine, or aggregate, these data to present follow-up results that can be used for 
making meaningful comparisons and program decisions. They might, for example, 
combine results across SDAs, across years, or across similar programs. Clearly labeled 
tables or graphs, and prose summaries should present relevant data in a concise way. 
Exhibit 3.2 provides examples of reporting formats that can be meaningful for follow-up 
data from States, individual SDAs (if the numbers in each group are sufficient), or 
combinations of SDAs in a State. 
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Exhibit 3.2 



Sample Report Formats for JTPA Follow-up Data 
Suggested Reports to SDAs 



1. Summaiy of SDA Data Compared to State Totals 

Is Difference 



SDA State Average Significant?* 

Sample Size 1000 600 yes 

Number of Respondents 900 450 yes 

Response Rate 90% 75% yes 

Follow-up E nployment Rate 60% 61% no 

Average Weekly Earnings $322 $307 no 

Average Weeks Worked 10.3 6.8 yes 

2. Comparison of Follow-up Data by SDA Contractors 

Is Difference 

Contractor A SDA Average Significant?* 



Sample Size 

Number of Respondents 
Response Rate 
Follow-up Employment Rate 
Average Weekly Earnings 
Average Weeks Worked 



3. Comparison of FoIIow-up Data by Type of Program within SDA 

Is Difference 

Program A SDA Average Significant?* 

Sample Size 

Number of Respondents 
Response Rate 
Follow-up Employment Rate 
Average Weekly Earnings 
Average Weeks Worked 



* refers to statistical significance 

« 
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Exhibit 3.2 (Continued) 



OTPA POSTPROGRAH FOLLOW-UP REPORT 
PY '87.- FOURTH QUARTER 

Table 1 orovldes statewide summary data on the postprogram •mplbyment and earnings 
exoerlences for several Title IIA subgroups. As the table Indicates, 4,157 Tltlt IIA 
adult terSlnees were Interviewed during this period. Sixty-eight percent (68X) werji 
employed during the thirteenth week following termination and were earning ;n avtrage 
weekly wage of $195.33. Termlnees reported an average of 9 weeks employment during tne 
13 week period following 3TPA participation and, for those reporting any employment, 
worked an average of 34 hours per week. 

In examining subgroups of respondents, women were as likely as men to be •inP]py;«*» 
were earning an average of $60 less per week than men. Dropouts were less likely to be 
employed thirteen weeks after JTPA, and earned about $20 less per week than non-dropouts 
when employed. Older workers (age 55 and over) experienced t!;^ioy"**veraae weekly 
earnings ($148.30). The employment and earnings experience of AFDC/WEOP recipients Is 
quite similar to the overall population of respondents. 

TA8LE 1 

SELECTED OUTCOMES 8Y CHARACTERISTICS OF JTPA TITLE IIA FOLLOW-UP RESPONDENTS 



Number of 



Percent 
Employed 



Average 
Weekly 



Average 
Weeks 



Average 
Weekly Hours 



Women 
Hen 


2.507 
1,650 


681 
68 


»r ILLUMJ 

$169.98 
229.52 


9.0 
9.2 


32.6 
35.4 


Dropouts 
Non-dropouts 


812 
3,345 


59 
70 


178.20 
198.93 


7.7 
9.4 


32.7 
34.0 


Minorities 
Whites 


619 
3,538 


51 
72 


186.14 
196.98 


7.0 
9.6 


33.5 
33.8 


Handicapped 
Non-Handicapped 


723 
3,434 


66 
68 


185.53 
197.25 


8.9 
9.1 


31.9 
34.1 


Age 55 and Over 
Under Age 55 


526 
3,631 


62 
69 


148.30 
201.79 


8.5 
9.2 


28.4 
34.5 


AFDC/WEOP Recipient 
Non-AFDC/WEOP 


1,060 
3,097 


65 
68 


189.11 
196.90 


8.9 
9.1 


33.9 
33.7 


Welfare Recipient 
Non-Welfare 


2,338 
1,819 


63 
73 


198.09 
193.06 


8.4 
9.7 


34.0 
33.5 


Total Respondents 


4,157 


68 


195.33 


9.1 


33.8 
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Exhibit 3.2 (Continued) 



Job Training Council 
13-Week Earnings and Employment Status by M^or Activity 



# of % Employed Avg. Wkly. Earnings 

Responses at 13WK of Those Employed 

Activity 

Classroom Training 159 67% $248.68 

Customized Traim'ng 73 82% $282.38 

OJT 89 81% $267.92 

Supplementary Work 28 46% $188.08 

Tuition Assistance 110 81% $316.11 

Job Search 51 69% $253.82 

TOTAL ALL RESPONDENTS 510 74% $272.02 
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Figure I: Enploywent Status by Age, Gender A Education; 

13 Ueeks After TerMination 
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Total (N=55e) 
Post H.S. (N=96) 
H.S. Grad. (N=358) 
Dropout (N=96) 
Fenale (N-325) 
Male (N=225) 
Youth (N:=95) 
Uelfare (N=148) 
Adult (N=3e7) 




5 
I' 



id 20 38 



Prepared by EnployMent 
and Training Institute 



40 50 60 

Percent 
Employed 



100 
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Jobville Follow-Up Results 
PrograM Year Ending June 38 « 1987 

Figure If: Average Ueeklij Gross Uage of Enployed by Age« 
Gender d Education; 13 Weeks After Temination 



388 T 



258 



288 



Dollars 158" 



188 



58 



8 




Adult Uelfare Youth Hale Fenale Dropout 
(N^7)(N=148) CN=95) (H=225) (H=325) (H=96) 



H.S. 
Grad. 
CN=358) 



Post Total 
H.S. (N=558) 
(N=96) 
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Limitations In the data. Despite the value of using follow-up information, even 
completely accurate information might not be meaningful. The significance of the data 
depends on the size of each group being examined. A small SDA that wants to evaluate 
different programs in terms of their follow-up employment rates might not be able to 
make reliable conclusions based on small numbers of terminees from each program, 
even if all terminees were followed up. 

• 

For example, suppose that in one SDA 60 terminees participated in classroom 
training programs, and 60 participated in OJT programs. If 50% (30) of those in 
classroom training were working for pay in week 13, then it would take 65% (39) of the 
60 in OJT for OJT to be judged significantly more effective than classroom training. A 
smaller difference could be attributed to random chance: it might have just happened 
that individual employers for classroom training terminees had business slowdowns. 

These limitations also apply to follow-up data that are not required by DOL--for 
example, responses to attitudinal questions. It is extremely important not to 
overinterpret the intense response of one or a small number of respondents. Program 
decisions must not be based on one terminee's opinion of a classroom training program 
(favorable or unfavorable); however, if all or a substantial number/percentage of 
terminees concur, the results might appropriately affect decisionmaking. 

In interpreting the follow-up data, especially from small samples, it is important to 
ask whether the participants in one program were different from those in another, in a 
way that might account for differences in follow-up results. For example, contractors 
serving harder-to-serve participants can be expected to have somewhat lower follow-up 
employment rates. They should not be penalized for reasonable deviations from average 
follow-up rates. 

Exhibit 3.3 presents the minimum numbers of follow-up terminees required in a 
group for percentage differences in follow-up employment rates between groups to be 
meaningful. Note that the numbers in the right column of the table refer to respondents 
(completed Interviews) only, not nonrespondents in the sample. 
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Minimum Sample Size for Differences In Follow-up Employment Rates 
Between Two Groups to be Meaningftii* 



Difference in % Employed 



Minimum Number of Respondents in Smallest 
Group for Difference to be Meaningful 



5% 

6 

8 
10 
12 
15 
20 
25 
30 
40 



540 
375 
211 
135 
94 
60 
34 
22 
15 
10 



'These numbers and percentages apply (o differences in employment rates only. Corresponding numbers for weekly earnings and 
weeks worked cannot be presented because only aggregate SDA data are available to DOL However, they can be calculated at the 
State and local levels, where individual SDA data are available. 



For example, if two training programs had follow-up employment rates of 75% 
and 55% respectively, there would be a 20% difference. This difference would be 
significant, and therefore meaningful and useful for program decisions, only if there were 
at least 34 follow-up respondents in each group. (Note in Exhibit 3.3 that a difference of 
20%, in the left hand of the table, requires at least 34 respondents in each g:oup.) For 
small SDAs, it may take a substantial time to compile the data needed for making 
reliable decisions, where differences are small: and programs may change during that 
period. On the other hand, States may be ab. . to support local decisionmaking by 
combining follow-up data from similar training programs across SDAs in the State. 
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Section 4 - Assuring the Accuracy and Validity of FoUow-up Data 

(VaUdation) 



This section presents validation in the context of JTPA follow-up data collection. 
It includes (1) an overview which discusses the importance of validation, the need for 
communicating validation activities and results to all involved in the follow-up process, 
and the relationship between validation and centralization or decentralization of follow- 
up; (2) a description of various validation methods; and (3) other issues related to 
validation. 



4.1 Overview of Validation 

Validation is essential in a performance management system like JTPA that 
rewards programs based on outcomes. To award incentives fairly and to make intelligent 
management decisions, it is critical that the follow-up data be accurate and that 
procedures for collecting the data be consistent and meet the Federal requirements. 
Although it is equally important for termination data to be accurate and valid (since they 
are also used for management decisions and to determine incentives), MIS records are 
typically available for documenting and monitoring termination outcomes. Follow-up 
data, however, depend on a respondent's self-report of employment status and earnings. 
Validation is especially needed to ensure consistency across SDAs in decentralized 
States, because locating and interviewing respondents can be done in different settings 
with different procedures and outcomes. Even in centralized States, there is a need to 
ensure consistency in the contact information provided by SDAs. 

This TAG has presented data collection procedures designed to maximize 
response rates and enhance the consistency and accuracy of the follow-up information 
gathered. While these procedures go a long way toward ensuring accurate data, 
additional steps can and should be taken to validate the follow-up data, from the 
beginning to the end of the process. The need for validation and its cost depends upon 
several factors, including the organization of the follow-up (whether the State or SDA 
collects the data), the extent cf the follow-up effort, and the size of the respondent 
population. Although DOL does not require validation of follow-up data. States should 
assess the need for validation in terms of these factors, and implement consistent 
procedures across SDAs that produce the most accurate follow-up data possible. 
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Deflning validation. For follow-up data collection, validation includes any and 
all activities undertaken to assess and assure the accuracy of the data collected, 
reported, and ultimately used for important management decisions* Broadly defined, 
validation includes all monitoring and quality control procedures designed to assure that 
appropriate and consistent follow-up methods are used. More narrowly defined, 
validation includes activities designed to verify, against additional sources of 
information, the accuracy and corisistency of the follow-up data itself. Both types of 
validation activities are relevant to insuring high quality follow-up data. 

Accuracy of the follow-up data refers to the correctness of the figures reported. 
Do the reported figures correctly capture each of the follow-up data items, including 
employment status, weekly earnings, and weeks worked? If a respondent is reported as 
"employed" at 13 weeks but was actually unemployed, then the data are inaccurate. 
Although data reported to DOL may contain some minimal error, it is important that the 
extent of error is small and consistent-that it does not vary among SDAs. 

Implications of centralization on need for validation. The need for validation, 
the extent of the validation effort required, and its expense will likely be greater if the 
responsibility for follow-up is decentralized. When each SDA performs its own 
follow-up, validation becomes particularly important because States must guard against 
inconsistent data collection and quality control procedures, as well as inaccuracy from 
biased reporting. States and SDAs within each State must be assured that all of the 
other SDAS have implemented the follow-up data collection procedures exactly and that 
the reported data accurately represent the employment experiences of terminees during 
the follow-up period. Validation should be sufficiently extensive to let each actor in the 
system rest easy. Decentralized States will need to implement a more extensive, 
centralized validation to guard against the inconsistencies and biases inherent in follow- 
ups conducted at the SDA level. More extensive validation will be more expensive as 
well. Adequate training, setup, supervision, and monitoring of follow-up procedures and 
gathering data from additional sources of information from SDAs will increase costs. 

On the other hand, when follow-up is centralized at the State level-regardless of 
whether interviewing is in-house or contracted out-validation activities can be more 
limited, and therefore less expensive. The State's liability in granting awards based upon 
SDAs' follow-up performance encourages reporting as accurately and consistently as 
possible to preclude complaints from SDAs based upon inequitable treatment. The 
primary validation required may be procedural such as supervising staff or monitoring 
outside contracts. Activities would include supervising data collection f\ctivities and 
monitoring data entry, records, and reports. Because a centralized C uses consistent 
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42 VaUdation Methods 



This section describes a variety of validation methods, which fall into two 
principal categories: (1) monitoring and supervising activities and (2) validating follow- 
up data against additional sources of information* 

Monitoring and Supervising Follow-up Data Collection 

Monitoring follow-up data collection is critical to ensuring that appropriate 
contacts are being made, that all necessary procedures are being followed correctly and 
consistently, and that adequate training and technical assistant*, are prcdded. 
Monitoring activities ensure that all steps in the follow-up process are conducted 
correctly, from beginnii. ^ to end: sampling, timing, locating and contacting appropriate 
participants, interviewing, coding and entering data, preparing reports, and transmitting 
information in a timely fashion among States, SDAs, contractors, and the DOL. The 
following procedures should be standard practice, whether the follow-up is centralized, 
decentralized, conducted in-house or contracted out. 

Specify a number or percent of interviews or records to monitor periodically. The 
percent will depend on the size of the follow-up effort. In a centralized follow-up or in a 
large SDA, a sample of at least 5-10% of the follow-up interviews for each interviewer 
should routinely be monitored. In a small SDA, at least 10 interviews for each 
interviewer should be monitored to ensure that they are being conducted properly. If 
the follow-up is contracted out, procedures such as these should be specified in the RFP 
and contract. 

• Listen in on interviews to make sure all questions are asked 
appropriately. Record the responses, and compare them to the 
responses recorded by the interviewer. Check to make sure 
ambiguities and inconsistencies were clarified in a neutral fashion. 
Provide interviewers with immediate feedback. Monitoring 
interviews in this way is an opportunity not only to validate, but to 
praise, instruct, and learn about effective interviewing techniques. 

• Review contact logs and completed interview records. Review the 
logs to ensure that adequate and timely attempts are made to reach 
terminees. Examine interview records for completeness and 
accuracy. 
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• Supervise and verify data entry. Double data entry (100% key 
verification) is recommended to ensure accuracy. 

• Check data against the DOL limits described in Section 2.6, 

• Establish additional MIS checks for internal consistency and reasonableness, 
based on local labor market conditions and common sense. Identify and 
resolve inconsistent or out-of-range responses-for example, set limits for 
hourly wages at $3.80 • $15.00, and for hours worked during the 13th week at 
0 -70. Compare paper records with MIS records to identify and resolve 
discrepancies-for example, large disparities between placement and follow-up 
data. 

• Verify the summaiy follow-up data that will be reported to the DOL against 
individual terminee records. This may involve comparison of paper records 
with MIS records. 

• Check the MIS software to ensure that it is accurately recording data and 
making appropriate edits. 

• Conduct periodic monitoring visits to ensure compliance if follow-up is 
done by SDAs or outside contractors. Follow-up should be an integral 
component of each State's monitoring activities. Monitoring should 
identify problems in procedures and data collected, and recommend 
corrective actions to ensure accuracy of the follow-up results. 

• If the above validation activities reveal a substantial number of discrepancies 
( > S% of the data examined), expand your validation activities to include a 
larger percentage of the follow-up sample. Monitoring or review of 100% of 
the interviews or records may, in fact, be warranted if discrepancies are very 
large. 

Validating Follow-up Data Against Additional Sources of Information 

Comparing follow-up data with information obtained from other sources is a 
useful way of validating the follow-up effort, particularly for decentralized States who are 
granting incentives based on the follow-up results. Several methods are appropriate. As 
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with monitoring activities, identify a specific percentage or number of the foUow-up 
sample to validate. 

• Re-interview respondents to assure they have been previously 
contacted and to verify their responses. Be sure to alert terminees 
during the initial interview that they might be called again by 
another person. 

• Contact employers, by phone or mail, to verify the follow-up 
employment and wage data provided by participants. If you contact 
employers, be sure to obtain signed permission from terminees to do 
so, (see Exhibit 2.4, Employer Release Form) and to establish 
cooperative relationships with local employers. When contacting 
employers, it is very important not to release confidential 
information a Jut participants or to disrupt or irritate employers 
who could be the source of future JTPA placements. It is also very 
important that information provided by employers NOT be used for 
JASR/WAPR reporting purposes in lieu of information provided by 
JTPA terminees. A sample employer verification form appears as 
Exhibit 4.1. 

• Asic additional questions and compare responses to MIS records to 
verify that correct participants have been interviewed and that 
responses are accurate. For example, have interviewers ask 
terminees for their birthdates, but do not provide this information 
to the interviewers. Then match the data to the MIS records. 

• In centralized States where SDAs do any independent follow-up, 
compare SDA data to State follow-up data. If two separate follow- 
ups are done by the State and SDA, participants should be informed 
that they may be called more than once and results can be 
compared. 

• Compare follow-up data to Unemployment Insurance (UI) wage 
records. Employers in all States are required to report the total 
wages paid to employees during each calendar quarter. This 
information, called the employee's UI Wage Record, can be 
compared to the wage information provided by participants as an 
approximate validation measure. The advantages of using UI 
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records are that it uses information that is already available, it is 
inexpensive, and it is relatively easy to do. The disadvantages are 
that the wage records do not include the specific JASR information 
on the 13th follow-up week, are not available until the second 
quarter after the JTPA follow-up period ends, and may not include 
all JTPA terminees (self-employed, out-of-State, and "underground" 
jobs may not be reported) or all income and employment for 
terminees (tips, second jobs). 
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Exhibit 4.1 



Sample Employment Verificatiou Form 

Company name 

Address . 

City, State, Zip 

Emp loyee^ 

Social Security Number 

Start date ^Currently working? Yes No 

Total number of weeks worked since start date 

Hourly wage since start date , 

Was Employee working the week of 

Number of Hours the Employee worked the week of__ ^Hrs. 

Employees Hourly Wage the week of $ 

Employees earnings before deductions from Tips $ ^Overtime $ 

Bonuses $ Commissions $ the week of 

Job Title 

Brief Job Description 



Please return to Camden County Employment & Training Center 
Thank you. 

*This form is used by the Camden County Employment and Training Center, Cainden, NJ. 
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43 Other Validation Issues 

Other validation issues that States and SDAs need to address include identifying 
and resolving discrepancies found when validating follow-up data, as well as the 
implications of invalid data. 

Identifying and Resolving Discrepancies 

Validating follow-up data with information obtained from other sources is a 
comparative process. Information provided by a respondent is compared with another 
supposedly accurate measure. If there are discrepancies, data should be changed to 
provide the most accurate information, if this is possible to determine. 

Unfortunately, the data used to validate a terminee's responses may be no better 
than the data provided by the terminee. Suppose the terminee reports earning $150.00 
during the 13th follow-up week. The employer reports that the terminee was paid 
$120.00 for that week. Who is correct? 

It is possible that both are correct. The terminee may have been paid only 
$120.00 by the contacted employer, but might have earned $30.00 in tips or at a second 
job. It is also possible that both are wrong. For these reasons, dcta should not be 
changed without recontacting the respondent, providing information about the 
discrepancy, and asldng for assistance in its resolution. 

s 

Comparisons with Unemployment Insurance (UI) records can also lead to the 
identification of apparent discrepancies. UI records may not match. It must be 
remembered that not all employers contribute to the UI pool, that individuals often earn 
tips or other income that are not reliably reported to UI, and that UI records are 
sometimes incorrect. 

Another type of validation involves reint^'.rviewing the respondent. Because 
human memory is imperfect, if you were asked, "How much money did you earn last 
week?" and a month later were asked the same question, you would most likely give two 
different numbers. (You might also believe that your original answer was more accurate 
since it was closer to the week in question.) 

Validation efforts will frequently lead to the identification of discrepancies. If the 
discrepancies are in only a small percentage of cases (i.e., fewer than 10%) and are 
small and not always in the same direction (i.e., systematic over-reporting of income), 
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they can be tolerated because they will not seriously distort aggregate statistics. 
However, unresolved discrepancies can seriously reduce the value of follow-up effort if 
any one of these conditions occurs: 

• more than a small percentage of cases have discrepancies 

• any of the discrepancies are large 

• the discrepancies are always in the same direction 

If respondents consistently under- or over-report their earnings, data will be 
biased and conclusions drawn from these data may not be meaningful. Validations that 
result in the detection of a consistent directional bias demand further action. The first 
thing to do is to ascertain that the source of the validating information is unbiased. As 
previously mentioned, UI records and employers may not include tips or other income. 
Recontacting respondents is essential to get more accurate employment and earnings 
information and to determine the source of any inaccu/acies. Once you know why 
incorrect data were provided, you can develop new procedures or modify old procedures 
to avoid recurrence of such problems. 

Large discrepancies may indicate misinterpretations or improper administration of 
the questions. Once again, the source of the validating information should be checked to 
make certain that critical components are not being excluded or that improper items are 
not being included. Recontacting respondent is again necessary to determine the source 
of the inaccuracies and to permit modification of old procedures or the development of 
new procedures to avoid recurrence of these problems. 

Implications of Invalid liata 

Random discrepancies have no more effect on the value of the follow-up data 
than sampling enor. However, systematic discrepancies, or bias, are more serious; they 
can lead to erroneous conclusions about program effectiveness. The detection of bias in 
the data demands that procedures be reviewed and changed to avoid this bias. Such 
information should be shared with the Department of Labor to permit proper 
interpretation of the employment and income data you provide them. 
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In addition, discrepancies identified during validation efforts must be 
communicated with the State, SDA, or contractor staff involved. Corrective action 
should be designed to eliminate the problems found during monitoring and validation. 
Appropriate corrective action will depend upon the unique situation of the State or SDA 
involved, and might relate to various components of the follow-up system, ranging from 
interviewer training, to procedures for locating terminees, to procedures for entering and 
analyzing the data. 

Regardless of the outcome of your validation efforts, the DOL should be 
informed of their results. It is very useful to have an indication of validity to interpret 
data properly. When you are unsure of the level of validity, it is difficult to draw 
meaningful conclusions and make recommendations based on these data. 
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Appendix A 

JTPA Annual Status Report and Instructions (1989 JASR) 

[NOTE: The PY 1990 JASR and Instructions should be inserted here when it becomes available.] 
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Purpose . The JTPA Annual Status Report (JASR) displays 
cumulative data on participation, termination, performance 
measures and the socio-economic characteristics of all 
terminees on an annual basis. The information will be used 
to determine levels of program service and performance 
measures. Selected information will be aggregated to provide 
quantitative program accomplishments on a local. State, and 
national basis. 

General Instructions . The Governor will submit: 1) a com- 
bined statewide JASR for Title III Formula and Discretionary 
National Reserve (Column D only); 2) for Title ii-A (Columns 
A-C) a separate JASR for each designated Service Delivery 
Area (SDA) and 3) a separate Statewide JASR summary of the 
SDA report data. (This Statewide JASR summary of individual 
SDA data should not be submitted by single-SDA States.) 
Grantees may determine whether the reports are submitted on 
JASR forms or as a computer printout, with data, including 
signature and title, date signed and telephone number, 
arrayed as indicated on the JASR form. if revisions are made 
to the JASR data after the reporting deadline, revised copies 
of the JASR should be submitted to DOL as soon as possible 
according to the required reporting procedures. Submittal of 
one or more JASRs with revised information for Total Partici- 
pants, Total Terminations and/or Total Program Costs (Federal 
Funds) usually will require submittal of a revised JSSR which 
includes the final quarter of the same program year. 

NOTE: For JASR reporting purposes. Title ll-A shall refer to 
programs operated with funds authorized under Section 202(a) 
of the Act or otherwise distributed by the Governor under 
Section 202(b)(3) (6%) of the Act— incent ive grants for 
service to the hard-to-serve and programs exceeding 
performance standards. (Concentrated Employment Programs 
(CEPs) should report total Title ii-A program expenditures of 
78% funds, special supplemental allocations, and 6% incentive 
grants.) Do not include data on (6%) funds authorized under 
Section 202(b)(3) for technical assistance. Participants and 
expenditures under Title I, Sections 123 (8%) and 124 (3%), 
and expenditures under Title II, Section 202 (b)(4) (5%) and 
any participants, if applicable, are likewise excluded from 
the JASR. 

NOTE: Participant and expenditure information under Title 
II-B, Summer Youth Employment and Training Program (SYETP) , 
is also excluded from the JASR. 

SDAs should not terminate from Title ii-A youths who 
participate in the Title II-B Summer Program unless they are 
not expected to return to Title ll-A for further employment, 
training and/or services. 

If these youths receive concurrent employment, training 
and/or services under both Titles ll-A and II-B, they 
are to be considered participants in both titles for 

(Rev. 2/90) 
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purposes of recording actual number of weeks 
participated, dollars expended, and other pertinent 
data . 

If, however, these youths do not receive Title li-A 
employment, training and/or services while 
participating in Title ll-B, this period is not to be 
included in the calculation of actual number of weeks 
participated in Title Il-A at Line 29, Column C. 

The reporting period begins on the starting date of each jtpA 
program year, as stated in Section 161 of the Act. Reports 
are due in the national and regional offices no later than 45 
days after the end of each program year. Two copies of the 
JASR are to be provided to: 

Employment and Training Administration 
U.S. Department of Labor , 
ATTN: TSVR — Rm . S-5306 
200 Constitution Avenue, N.W. 
Washington D.C. 20210 

At the same time an additional copy of the jasr is to be 
provided to the appropriate Regional Administrator for 
Employment and Training in the DOL regional office that 
includes the state in which the JTPA recipient is located. 

3. Facsimile of Form . See the following page. 

^- Instructions for Com pleting the JTPA Annual stat us Report 
(JASR) . ' 

3. State/SDA Name, Number and Address 

Enter the name and address of the state agency that will 
administer the grant recipient's program (Title m 
report). Enter the name, ETA assigned SPA nmabe r and 

address of the designated SDA subrecipient , as 

appropriate (Title ii-a report). 

b . Report Period 

Enter in "From" the beginning date of the designated JTPA 
program year and enter in "to" the ending date of that 
program year . 

Signature and Title (at bottom of the page) 

The authorized official signs here and enters his/her 
title. 



ERIC 



105 




L'.S, DEPARI>l£NT OF LABOR 
E^nployment and Training Ministration 

JIM 

MHMO. SOam MKKt 



a, STATt;'£li\ NAME ASD ADDRESS 



REPORT PERIOD 



PHON 



TO 





Z. PARnCIMTION AND TEFNZNATIGN 
SUMMARY 


TbUl 
Adults 


Mults 


Youth 


Dialocatad 






(A) 


(B) 


(C) 


(D) 


A. TOTAL PARnCIFMirS 










B. Tom mmwncHs 










1. D)ttr^ Uiaubtiditad fiifsloymant 










a. Also Attainad Any Youth Ehf)loyability Ehhancmnt 










2. Youth E>nployability Ehhanccaant Tenninationa 










a. Attainad PIC-Recognized Youth Eh9)loymtnt 
Gontpetencies 










b. Ocxnpletad Program Objactives (14-15 yaar olds) 












3. All Other Itetmirationt { 




1 






• 

t 
t 

m 
•I 


II. TGmiNEES POtPOIWANCE MEASURES 
INPOWMnON 




Ic SIGNKIUIC AND Tni£ 



d. DAIE SIGNED 



t. TGLE. NO. 



E1A 1580 (Juro IMS) 



BEST COPY AVAILABLE 



a • STA7E;^S0A liME AND ADDRESS 


REPORT PERICX) 




FICM 


TO 



i 


II. TEFMINEZS PERFOmMCE MEASURES 


lt>tal 
Mults 


Multt 
(Helfare) 


youth 


DitlccattH 


t 

2 






(A) 


(B) 


(C) 


(D) 


19 


• 


Limited Diglish Language Proficiency 










20 


# c 
H * 


Handicapped 










21 


« O 

a 


Offender 










22 


£ 

^ O 


Reading Skills Below 7th Grade Level 










23 


O H 


Long-Term AFDC Recipient 






24 


it 


unemployment Oompensation Claimant 










25 


Uiemployed: IS or n3re Weeks of Prior 26 Weeks 










26 




M>t in labor FDrce 










27 


«< € 

m M 

il 

-I 


Welfare Grant Type: AFDC 










28 


GVICA 










29 


Average Weeks Participated 










30 


Average H3urly Wage at TCrmircition 










31 


Ibtal Program Oosts (FMeral Funds) 








32 


Tt>tal Available Federal Fwds ^ 









III. POLLCW-OP INPOPMATION 



33 


Bnployment Rate (At Pbllow-up) 






34 


Average Weekly Earnings of Dnployed (At Ftollow-up) 






35 


Average Number of Weeks UDrked in 
Fbllow-up Period 






36 


Sample Size 






37 


Re^onse Rate 







IV. YOUTO BIPLOWair CCMPCTENCy ATTAlrmrr INPOfWATION 




38 


AtUiiMd Any Oonqptttncy Area 








39 


Pre-EJnployment/Mbrk Maturity Skills 




40 


Baiic Blucatlon Skilli 




41 

mm 


Job specific Skill! 





Mgt 3 of 2 Pttges 



ERIC 



107 



m 8S80 (June 1988) 



-3- 



d . Date Signed 

Enter the date the report was signed by the authorized 
official. 

e. Telephone Number 

Enter the area code and telephone number of the 
authorized official. 

5. General Information . For purposes of the JASR, the Total 
Adults and Adults (Welfare) columns will include terminees 
age 22 years and older. Thus, the column breakouts are based 
strictly on age rather than on program strategy. The youth 
column will include terminees who were age 14-21 at the time 
of eligibility determination. The Dislocated Workers column 
may include adults and youth, as applicable. 

Unless otherwise indicated, data reported on characteristics 
of terminees should be based on information collected at the 
time of eligibility determination. 

CHARACTERISTICS INFORMATION OBTAINED ON AN INDIVIDUAL AT THE TIKE 
OP ELIGIBILITY DETERMINATION FOR THE RECIPIENT'S JTPA PROGRAM 
SHOULD NOT BE UPDATED WHEN THE INDIVIDUAL TERMINATES FROM THE 
JTPA PROGRAM. 

COLUMN HEADINGS 

Column A Total Adults 

This column will contain an entry for each appropriate item for 
all adult participants in Title II-A only. 

Column B Adults (Welfare) 

This column will contain an entry for each appropriate item for 
adult participants in Title II-A who were listed on the welfare* 
grant and were receiving cash payments under AFDC (SSA Title IV). 
General Assistance (State or local government), or the Refugee 
Assistance Act of 1980 (PL 96-212) at the time of JTPA 
eligibility determination. For reporting and performance 
standards purposes, exclude those individuals who receive only 
SSI (SSA Title XVI) from entries in Column B. 

NOTE: Column B is a sub-breakout of Column A; therefore. Column 
B should be less than or equal to Column A for each line entry. 

Column C Youth 

This column will contain an entry for each appropriate item for 
all participants, aged 14 - 21, in Title II-A only. 
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Column D Dislocated Workers 

This column will contain an entry for each appropriate item for 
all participants in Title III Formula and Discretionary National 
Reserve who were determined to be eligible dislocated workers. 

NOTE: Columns A, B, and C apply to Title II-A only. Column D 
applies to Title III orily. All information regarding a given 
participant must be entered in the same column, e.g., Column C 
for a youth in Title II-A. 

The sum of the entries (all SDAs in a State) in Columns A and C, 
Item I. A., Total Participants, of the JASR should equal the entry 
in Column A, Item III.A.l., SDA Participants, of the JSSR, for 
the same recipient, that includes the final quarter of the same 
program year. The entry in Column D, Item I. A. of the Statewide 
JASR for Title III should be the sum of the entries in Columns B 
and C, Item III. A. of the JSSR, for the same recipient, that 
includes the final quarter of the same program year for the same 
grant . 

The sum of the entries (all SDAs in a State) in Columns A and C, 
Item I.B., Total Terminations, of the JASR should equal the entry 
in Column A, Item III.B.I., SDA Terminations, of the JSSR, for 
the same recipient, that includes the final quarter of the same 
program year. The entry in Column D, Item l.B. of che statewide 
JASR for Title III should be the sum of the entries in Columns B 
and C, Item III.B. of the JSSR, for the same recipient, that 
includes the final quarter of the same program year for the same 
gr?int . 

SECTION I - PARTICIPATION AND TERMINATION SUMMARY 



Section I displays the program's accomplishments in terms of the 
total cumulative number of participants in the program and the 
number and types of terminations from the program, as of the end 
of the reporting period. 

Entries for items I. A. and l.B. are cumulative from the beginning 
of the program year through the end of the reporting period. 

Item I. A. Total Participants 

Enter by column the total number of participants who are or were 
receiving employment, training or services (except post- termina- 
tion services) funded under that program title through the end of 
the reporting period, including both those on board at the begin- 
ning of the designated program year and those who have entered 
during the program year. if individuals receive concurrent 
employment, training and/or services under more than one title, 
they are to be considered participants in both titles for 
purposes of recording actual number of weeks participated, 
dollars expended, and other pertinent data. 
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"Participant" means any individual who has: (1) been determined 
eligible for participation upon intake; and (2) started receiving 
employment, training, or services (except post- termination 
services) funded under the Act, following intake. Individuals 
who receive only outreach and/or intake and initial assessment 
services or postprogram follow-up are excluded. 

participants who have transferred from one title to another, or 
between programs of the same title, should be recorded as 
terminations from the title or program of initial participation 
and included as participants in the title or program into which 
they have transferred, unless they are to be considered 
concurrent participants in both titles or programs. 

Item I.B. Total Terminations 



Enter by column the total number of participants terminated after 
receiving employment, training, or services (except post- termina- 
tion services) funded under that program title, for any reason, 
from the beginning of the program year throuc-h the end of the 
reporting period. This item is the sum of Ite.-ns T.B.I, through 
I • B • 3 • 



Termination" means the separation of a participant from a given 
title of the Act who is no longer receiving employment, training, 
or services (except post- termination services) funded under that 
title. NOTE: Individuals may continue to be considered as par- 
ticipants for a single period of 90 days after last receipt of 
employment and/or training funded under a given title. During 
the 90-day period, individuals may or may not have received 
services. For purposes of calculating average weeks 
participated, this period between "last receipt of employment 
and/or training funded under a given title" and actual date of 
termination is defined as "inactive status" and is not to be 
included in Line 29. 

Individuals who initially participate in Title III Formula funded 
activity and subsequently participate in Title ill Discretionary 
National Reserve funded activity (and conversely) , FOR THE 
COMPLETION OF THE INITIALLY DETERMINED TRAINING OBJECTIVE, may be 
considered to be concurrent participants in both Title III 
programs. The type of termination determined for the subsequent 
program also should be recorded for the initial program for such 
participants. 

For purposes of calculating average weeks participated for such 
concurrent Title III program participants, the period between 

last receipt of employment and/or training funded under a given 
Title III program" (i.e.. Formula or Discretionary National 
Reserve) and actual date of termination from that Title ill 
program is defined as "inactive status" and is not to be included 
in Line 29. 
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Item I.B.I. Entered Unsubsidized Employment 



Enter by column the total number of participants who, at termina- 
tion, entered full- or part-time unsubsidized employment through 
the end of the reporting period. Unsubsidized employment means 
employment not financed from funds provided under the Act and 
includes, for JTPA reporting purposes, entry into the Armed 
Forces, entry into employment in a registered apprenticeship 
program, and terminees who became self-employed. 

Item I.B.I. a. Also Attained Any Youth Employability Enhancement 

Enter the total number of youth who (1) entered unsubsidized 
employment. Item I.B.I. , and (2) also attained any one of the 
five youth employability enhancements (as enumerated in the 
instructions for Item I.B.2. below and defined in Appendix C) . 
This item is a sub-breakout of item I.B.I. 

Item I.B.2. Youth Employability Enhancement Terminations 

Enter the total number of youth who were terminated under one of 
the Youth Employability Enhancements through the end of the 
report period. "Youth Employability Enhancement" means an out- 
come for youth, other than entered unsubsidized employment, which 
is recogn.^iid as enhancing long-term employability and contribu- 
ting to tiie potential for a long-term increase in earnings and 
employment. Outcomes which meet this requirement shall be 
restricted to the following: (1) Attained PIC-Recognized Youth 
Employment Conpetencies (two or more); (2) Entered Non-Title II 
Training; (3) Returned to Pull-Time School; (4) Completed Major 
Level of Education; or (5) Completed Program Objectives (14-15 
year olds) . 

NOTE: For reporting purposes, a youth shall not be counted in 
Item I. B.2. if s/he entered unsubsidized employent , and shall be 
counted in only one of the five categories enumerated above, even 
though more than one outcome may have been achieved. 

Item I.B.2.a. Attained PIC-Recognized Youth Employment 

Competencies 

Enter the total number of youth who, at termination, have demon- 
strated proficiency as defined by the PIC in two or more of the 
following three skill areas in which the terminee was deficient 
at enrollment: pre-employment/wor k maturity, basic education, or 
job-specific skills. Competency gains must be achieved through 
program participation and be tracked through sufficiently 
developed systems that must include: quantifiable learning 
objectives, related curricula/training modules, pre- and post- 
assessment, employability planning, documentation, and certifi- 
cation. This item is a sub-breaKout of item I.B.2. 
During py '88, competency attainment in one skill area will meet 
this definition. The entry for Item I.B.2.a. must be equal to or 
smaller than the entry for Line 38 in PY 1988 and PY 1989. 

Rev. 9/89 
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NOTE: Terminees who have attained a competency in basic 
education skills and/or job specific skills through training 
funded under 8% prograirs and/or cooperative agreements may be 

counted in Item I.B.2.a. provided such training was for 
completion of a training objective initially determined while in 
a youth employment competency sys^-em operated under 78% funds. 

Appendix B defines the minimal structural and procedural elements 
of a sufficiently developed youth employment competency system as 
well as the minimal requirements for ensuring consistency in the 
reporting of competency attainment in the pre~employment/work 
maturity skill area. 

Item I.B.2.b. Completed Program Objectives (14-15 year olds) 

Enter the total number of youth (ages 14 and 15 only) who, at 
termination, had completed a program objective. This item is a 
sub-breakout of Item I.B.2. 

NOTE: For Column C, (1) Item I.B.2,b. cannot be greater than 
Item II. 3., (2) the sum of Items I.B.2.a. and I.B.2.b. cannot be 
greater than Item l.B.2. and (3) Item l.B.l. plus item I.B.2. 
plus Item I.B.3. must equal Item I.B. 

Item I.B. 3. All Other Terminations 

Enter by column the total number of participants who were termi- 
nated for reasons other than those in Items l.B.l. and I.B. 2., 
successful or otherwise, through the end of the reporting period. 
Include intertitle transfers here , see notes at item i.e. 

SECTION II - TERMINEE PERFORMANCE MEASURES INFORMATION 

Section ll displays performance measures/parameters information. 
As indicated previously, data reported on characteristics of 
terminees should be based on information collected at time of 
eligibility determination unless otherwise indicated. 

Governors may develop any participant record which meets the 
requirements of Section 629.35(c) and (d) of the JTPA 
regulations. The DOL/ETA Technical Assistance Guid e! The JTPA 
Participa nt Record , dated May 1983, may be used as a refere nce. 

Line item Definitions and Instructions 
Sex 

Line 1 Male 
Line 2 Female 

Distribute the terminees by column according to Sex. The sum of 
Lines 1 and 2 in each column should equal item I.B. in that 
column . 
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Age 






Line 


3 


14-15 


Line 


4 


16-17 


Line 


5 


18-21 


Line 


6 


22-29 


Line 


7 


30-54 


Line 


8 


55 and over 



Distribute the terminees by column according to Age. The sum of 
Lines 3 through 8 in each column should equal Item l.B. in that 
column . 

Education Status 

Line 9 School Dropout 
Line 10 Student 

Line 11 High School Graduate or Equivalent (No Post-High 
School) 

Line 12 Post-High School Attendee 

Distribute the terminees by column according to Education 
Status. The sum of Lines 9 through 12 in each column should 
equal Item l.B. in that column. 

Family status 

Line 13 Single Head of Household with Dependent(s) Under Age 
18* 

Enter the total number of terminees by column for whom the above 
Family Status rlassif icat ion applies. 

Race/Ethnic Group 

Line 14 White (Not Hispanic) 
Line 15 Black (Not Hispanic) 
Line 16 Hispanic 

Line 17 American Indian or Alaskan Native 
Line 18 Asian or Pacific Islander 

Distribute the terminees by column according to the Race/Ethnic 
Groups listed above. For purposes of this report, Hawaiian 
Natives are to be recorded as "Asian or Pacific islander". The 
sum of Lines 14 through 18 in each column should equal Item I.E. 
in that column . 
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Other Barri^irs to Employment 

Line 19 Limited English Language Proficiency 
Line 20 Handicapped 
Line 21 Offender 

Line 22 Reading skills Below 7th Grade Level 
Line 23 Long-Term AFDC Recipient 

Enter the total number of terminees by column for whom each of 
the above Other Barriers to Employment apply, 

U. C. status 

Line 24 Unemployment Compensation Claimant 

Enter the total number of terminees by column for whom the above 
Unemployment Compensation Status classification applies. 

Labor Force Status 

Line 25 Unemployed: 15 or More Weeks of Prior 26 Weeks 
Line 26 Not in Labor Force 

Enter the total number of terminees by column for whom each of 
the above Labor Force status classifications apply. 

Welfare Grant Information 

Line 27 Welfare Grant Type: AFDC 
Line 28 GA/RCA 

Distribute by column the total number of adult and youth welfare 
terminees who, at eligibility determination, were listed on the 
welfare grant and were receiving ca&h payments under AFDC (SSA 
Title IV), GA, General Assistance (State or local government) or 
RCA (Refugee Cash Assistance) under the Refugee Assistance Act 
of 1980 (PL 96-212). If a welfare recipient terminee received 
AFDC cash payments, include such terminee on Line 27. A welfare 
recipient terminee who received cash payments under GA and/or 
RCA, but nojt AFDC, should be included on Line 28. ".he sum of 
Lines 27 and 28 in Column B, Adults (Welfare), should equal item 
I.B. in that column. The sum of Lines 27 and 28 in Column C, 
Youth, should be the same as or less than item l.B. in that 
column . 

Other Program Information 

Line 29 Average Weeks Participated 

Enter by column the average number of weeks of participation in 
the program for all terminees. Weeks of participation include 
the period from the date an individual becomes a participant in a 
given title through the date of a participant's last receipt of 
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employment and/oi training funded under that title. Exclude the 
single period of up to 90 days during which an individual nay 
remain in an inactive status prior to termination. Tine in 
inactive sta .us for all terninees should not be counted toward 
the actual number of weeks participated, inactive status is 
defined as that period between "last receipt of employment and/or 
training funded under a given title" and actual date of 
termination. See note at Item I.B. 

To calculate this entry: Count the number of days participated 
for each terminee, including weekends, from the start date of 
his/her participation in the title until his/her last receipt of 
employment and/or training under that title. For those who 
receive services only, use date of last receipt of such 
services. Divide this result by 7. This will give the number of 
weeks participated for that terminee. sum all the terminees' 
weeks of participation and divide the result by the number of 
terminees, as entered (by column) in Item I.B. This entry 
should be reported to the nearest whole week. 

Line 30 Average Hourly Wage at Termination 

Enter by column the average hourly wage at termination for the 
total number of terminees in Item l.B.l. 

To calculate this entry: Sum the hourly wage at termination for 
all the terminees shown in Item l.B.l. Divide the result by the 
number of terminees shown in Item l.B.l. 

Hourly wage includes any bonuses, tips, gratuities and 
commissions earned. 

Line 31 Total Program Costs (Federal Funds) 

Enter the total accrued expenditures, through the end of the 
reporting period, of the funds allocated to SDAs under Section 
202 (a) of the Act or otherwise distributed by the Governor to 
SDAs under Section 202(b)(3) — incentive grants for services to 
the hard-to-serve and programs exceeding performance standards — 
for Title ii-A programs in Columns A and C (includes costs of 
services to participants aged 14-21), as appropriate, for all 

participants served. Do not include expenditures of funds 

authorized under Section 202(b)(3) for technical assistance. 
Exclude expenditures under Title i, Sections 123 (8%) and 124 
(3%) and Title ll Section 202(b)(4) (5%). Enter the total 
accrued expenditures of Title ill funds received by the Governor 
under Section 301 of the Act in Column D only, for all Title iii 
participants served through the end of the repor ting~per iod . 
Include expenditures of Federal funds only, both Formula and 
Discretionary National Reserve. 

NOTE: Entries will be nade to the nearest dollar. Negative 
entries are not acceptable. The JASR program cost data will be 
compiled on an accrual basis. If the recipient's accounting 
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records are not normally maintained on an accrual basis, the 
accrual information should be developed through an analysis of 
the records on hand or on the basis of best estimates. 

The sum of the entries in Columns A and C, Line 31, Total Program 
Costs, of the JASR (i.e., total for the State's SDAs under Title 
II-A) should equal the entry in Column A, Item I.A.I. , SDA Total 
Program Expenditures, of the JSSR, and the sum of the entries 
(all SDAs in a State) in Column C, Line 31 of the JASR should 
equal the entry in Column A, Item ii. of the JSSR, for the same 
recipient, that includes the final quarter of the same program 
year. The entry in Column D, Line 31, of the Statewide JASR for 
Title III should be th>? sum of the entries in Columns B and C, 
Item I. A. of the JSSR, for the same recipient, that includes the 
final quarter of the same program year for the same grant. 

Line 32 Total Available Federal Funds 

Enter the total Federal funds available for the Title ll-A or 
Title III program described on this report including (1) unex- 
pended funds carried over from previous program years, (2) funds 
allocated or awarded for this program year, and (3) any realloca- 
tion that increased or decreased the amount of funds available 
^ ^ * • ? * . ^ ^ ^ this 

reporting period. Enter 
all Title ii-A funds (Adults and Youth) in Column A and all Title 
III funds in Column D. Title II-A funds include those allocated 
to the SDA by the Governor under Section 202(a) of the Act, as 
well as incentive grants for services to the hard-to-serve and 
for programs exceeding performance standards under Section 
202(b)(3). Exclude funds authorized under Section 202(b)(3) (6%) 
for technical assistance to SDAs and funds received for activi- 
ties under Sections 123 (84) and 124 (3%) and Section 202(b)(4) 
ini /UN '^^^^^ ^""^2 include all Formula amounts under Section 

301(b) and all Discretionary National Reserve awards under 
Section 301 (a) . 

SECTION III - FOLLOW-UP INFORMATION 

Section III displays information based on follow-up data which 
must be collected through participant contact to determine an 
individual's labor force status and earnings, if any, during the 
13th full calendar week after termination and the number of weeks 
s/he was employed during the 13-week period. Follow-up data 
should be collected from participants whose 13th full calendar 
week after termination ends during the program year (the 
follow-up group) . Thus, follow-up will be conducted for 
individuals who terminate during the first three quarters of the 
program year and the last quarter of the previous program year. 

Follow-up data will be collected for the following terminees: 
Title ll-A adults, adult welfare recipients, and Title ill 
dislocated workers (Columns A, B, and D) . No follow-up 
information is required for Title li-A youth (Column C) . 
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The procedures used to collect the follow-up data are at the 
discretion of the Governors. However, in order to ensure 
consistency of data collection and to guarantee the quality of 
the follow-up information, follow-up procedures must satisfy 
certain criteria. (See the Follow-up Guidelines included in 
these JASR instructions, Appendix A.) 

NOTE: Every precaution must be taken to prevent a "response 
bias" which could arise because it may be easier to contact par- 
ticipants who were employed at termination than those who were 
not and because those who entered employment at termination are 
more likely to be employed at follow-up. Special procedures have 
been developed by which SDAs and States can monitor response 
bias. If your response rates for those who were and were not 
employed at termination differ by more than 5 percentage points, 
the follow-up entries for the JASR must be calculated using the 
"Worksheet for Adjusting Follow-up Performance Measures* in the 
Follow-up Technical Assistance Guide. if the response rates 
differ by 5 percentage points or less, the following instructions 
for completing Lines 33-35 may be used. 

Line 33 Employment Rate (At Follow-up) 

Enter by column the employment rate at follow-up. 

Calculate the employment rate by dividing the total number of 
respondents who were employed (full-time or part-time) during the 
13th full calendar week after termination by the total number of 
respondents (i.e., terminees who completed follow-up 
interviews). Then multiply the result by 100. This entry should 
be reported to the nearest one decimal (00.0). 

Line 34 Average Weekly Earnings of Employed (At Follow-up) 

Enter by column the average weekly earnings of those employed 
(full-time or part-time) at follow-up. 

Calculate the (before-tax) average weekly earnings by multiplying 
the hourly wage by the number of reported hours for each 
respondent employed at follow-up; and, if appropriate, add tips, 
overtime, bonuses, etc. Divide the sum of weekly earnings for 
all respondents employed during the 13th full calendar week after 
termination by the number of respondents employed at the time of 
follow-up. Respondents not employed at follow-up rre not 
included in this average. This entry should be reported to the 
nearest whole dollar. *~ 

Weekly earnings include any wages, bonuses, tips, gratuities, 
commissions and overtime pay earned. 

Line 35 Average Number of Weeks Worked in Follow-up Period 

Enter by column the average number of weeks worked. 
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To calculate the average number of weeks worked (full-time or 
part-time) , divide the sum of the number of weeks worked during 
the 13 full calendar weeks after termination for all respondents 
who worked, by the total number of all respondents, whether or 
not they worked any time during this 13-week follow-up period. 
This entry should be reported to the nearest one decimal (00.0). 

Line 36 Sample Size 

Enter by column the size of the actual sample selected to be con- 
tacted for follow-up. (For Title ill only, a statewide sample of 
dislocated workers must be selected. For Title ll-A, i.e., total 
adults and adult welfare recipients, SDA samples must be 
selected . ) 

NOTE: If oversampling was used, the sample size should include 
all those selected, not just the required minimum sample size. 
Those deceased or severely incapacitated to the point of being 
unable to respond at follow-up may be excluded from the sample 
size. 

Line 37 Response Rate 

Enter by column the overall response rate, i.e., the percentage 
of complete surveys obtained. 

To calculate the overall response rate, divide the number of 
terminees with complete follow-up information by the total number 
of terminees included in the follow-up sample (Line 36) and 
multiply by 100. This entry should be reported to the nearest 
whole percent. 

NOTE: Complete follow-up information consists of substantive 
answers to the required follow-up questions and may not include 
don't know", "no answer" or "don't remember". 

SECTION IV - YOUTH EMPLOYMENT COMPETENCY ATTAINMENT INFORMATION 

Section IV displays information relevant to youth employment com- 
petency attainment as defined by the PIC. Regardless of termina- 
tion type, the following data represent the total cumulative 
number of individuals that attained a youth employment competency 
in any of the three skill areas and the numbers of individuals 
who attained a competency in (1) pre-employment/wor k maturity, 
(2) basic education and/or (3) job specific skills. 

NOTE: Terminees who have attained a competency in basic 
education skills and/or job specific skills through training 
funded under 8% programs and/or cooperative agreements may be 

counted in Section IV provided such training was for completion 
of a training objective initially determined while in a youth 
employment competency system operated under 78% funds. 
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Appendix B defines the mininal structural and procedural elements 
of a sufficiently developed youth employnient competency system as 
well as the minimal requirements for ensuring consistency in the 
reporting of competency attainment in the pre-employment/ work 
maturity skill area. 

Line ?Q Attained Any Competency Area 

Enter in Column C the total unduplicated number of youth termi- 
nees who were enrolled in a youth employment competency component 
and who attained a competency in at least one skill area. 

NOTE: Lines 39-41 are not sub-breakouts of Line 38 because one 
individual may attain several competencies and may be recorded on 
more than one of Lines 39-41. That individual may be recorded 
only once on Line 38, thus, the sum of the entries for Lines 
39-41 must be equal to or greater than the entry for Line 38. 

Line 39 Pr e-Employment/Wor k Maturity Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the pre-employment/wor k maturity skill area. 

Line 40 Basic Education Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the basic education skill area. 

Line 41 Job Specific Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the job specific skill area. 

NOTE: An entry of "O" on any of Lines 39-41 may indicate that 
the PIC has determined that a specific skill area is not 
necessary to become employment competent in their local labor 
mar ket . 
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Follow-up Guidelines 

To ensure consistent data collection and as accurate information 
as possible, procedures used to obtain follow-up information must 
satisfy the following criteria: 

** participant contact should be conducted by telephone or 
in person. Mail questionnaires may be used in those 
cases where an individual does not have a telephone or 
cannot be reached. 

* Participant contact must occur as soon as possible after 
the 13th full calendar week after termination but no 
later than the 17th calendar week after termination. 

* Data reported are to reflect the individual's labor force 
status and earnings during the 13th full calendar week 
after termination and the number of weeks s/he was 
employed throughout the 13-week period after termina'-ion. 

* Interview questions developed by DOL (see following 
Exhibit) must be used to determine the follow-up 
information reported on the JASR. Respondents must be 
told that responding is voluntary and that information 
provided by them will be kept confidential. Other 
questions may be included in the interview. Attitudinal 
questions may precede DOL questions, but questions 
related to employment and earnings must follow. 



120 



-16- 
EXHIBIT 



Minimum Postprogram Data Collection Questions 

I want to ask you about the week starting on Sunday, , 

and ending on Saturday, , which was (last 

week/two/three/four weeks ago) . 

1. Did you do any work for pay during th it week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour ii. that week? 

Dollars per hour 

4. How much extra, if any, did you earn in that week from 
tips, overtime, bonupes, commissions, or any work you did 
on the side, before deductions? 

Dollars 



Now I want to ask you about the entire 13 weeks from Sunday. 
, to Saturday, . 

5. Including the week we just talked about, how many weeks 
did you work at all for pay during the 13-week period? 

Weeks [Go to end] 



ALTERNATIVE QUESTIONS 
If answered "NO" to Question 1; 

Now I want to ask you about the entire 13 weeks from Sunday. 
, to Saturday, , ' 

6. Did you do any work for pay during that 13-week period? 

Yes [Go to 7] 

No [Go to end] 

7. How many weeks did you do any work at all for pay durina 
that 13-week period? ^ 
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* Attempts must be made to contact all individuals unless 
tenninee populations are large enough to use sampling. 

" At least six attempts may need to be made to contact 

enough individuals in the follow-up group to obtain the 
required response rate. 

* For each SDA (Title II-A) or combined Statewide (Title 
III Formula and National Reserve) report (JASR), minimum 
response rates of 70% are required for each of the 
following six groups: among adults, those who entered 
employment at termination and those who did not enter 
employment at termination; among welfare recipients, 
those who entered employment at termination and those who 
did not enter employment at termination; and among 
dislocated workers, those who entered employment at 
termination and those who did not enter employment at 
termination. The response rate is calculated as the 
number of terminees with complete follow-up information 
divided by the total number of terminees included in the 
group eligible for follow-up. 

Sampling Procedures 

Where sampling is used to obtain participant contact information. 
It is necessary to have a system which ensures consistent random 
selection of sample participants from all terminees in the group 
requiring follow-up. 

" No participant in the follow-up group may be arbitrarily 
excluded from the sample. 

Procedures used to select the sample must conform to 
generally accepted statistical practice, e.g., a table of 
random numbers or other random selection techniques must 
be used. 

The sample selected for contact must meet minimum sample 
size requirements indicated in Table 1. 

The use of sampling will depend on whether the terrainee 
populations are large enough to provide estimates which meet 
minimum statistical standards. If the number of terminees for 
whom follow-up is required is less than 138, sampling cannot be 
used. In such cases attempts must be made to contact all the 
appropriate terminees. 

Minimum Sample Sizes for Follow-up 

To determine the minimum number of terminees to be included in 
the follow-up sample, refer to Table 1 in the following 
Instructions. Find the row in the left-hand column that contains 
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the planned number of terminees for each of the groups requiring 
follow-up: adults, welfare recipients and dislocated workers. 
The required minimum sample size is given in the middle column of 
that row. The last column gives sampling percentages that will 
assure that the minimum sample size is obtained. 

NOTE: The welfare recipients in the adult sample may be used as 
part of the welfare sample. In this case, an additional number 
of welfare recipients must be randomly selected to provide a 
supplemental sample large enough to meet the same accuracy 
requirements as other groups requiring follow-up. To determine 
the minimum size of this supplemental welfare sample, find the 
row in the left-hand column of Table 1 that contains the planned 
total number of welfare recipients requiring follow-up. From the 
corresponding entry in the middle column, subtract the number of 
welfare recipients included in the adult sample. The remainder 
represents the minimum size of the supplemental sample of welfare 
recipients required for contact. 
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Table 1 

MINIMUM SAMPLE SIZES FOR POLLOW-UP 

Number of Terminees Minimum Sample Sampling 
in Follow-up Population size Percentaq 



1 - 137 


All 


100 


138 - 149 


137 


94 


150 - 159 


143 


92 


160 - 169 


149 


89 


170 - 179 


154 


87 


180 - 189 


159 


85 


190 - 199 


164 


84 


200 - 224 


175 


82 


225 - 249 


185 


78 


250 - 274 


194 


74 


275 - 299 


202 


71 


300 - 349 


217 


67 


350 - 399 


229 


62 


400 - 449 


240 


57 


450 - 499 


250 


53 


500 - 599 




50 


600 - 749 


282 


44 


750 - 999 


302 


38 


1,000 - 1,499 


325 


30 


1,500 - 1,999 


338 


22 


2,000 - 2,999 


352 


17 


3,000 - 4,999 


364 


12 


5,000 or more 


383 


7.3 
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Correcting for Differences in Response Rates 

Different response rates for those terminees who entered employ- 
ment at termination and those who did not are expected to bias 
the performance estimates because those who entered employment at 
termination are more likely to be employed at follow-up. It is 
assumed that those who were employed at termination are easier to 
locate than those who were unemployed because the interviewer has 
more contact sources (e.g., name of employer). The resulting 
response bias can artificially inflate performance results at 
follow-up. 

To account for this problem, separate response rates should be 
calculated for those who were employe<^ at termination and for 
those who were not. These separate response rates should be 
calculated for three groups: all adult II-A terminees, welfare 
recipients and Title III terminees. 

For each group, if the response rates of those employed at 
termination and those not employed differ by more than 5 percen- 
tage points, then the "Worksheet for Adjusting Follow-up 
Performance Measures" in the Follow-up Technical Assistance Guide 
must be used to correct the follow-up measures for that group. 



Appendix B 

PIC-REC0GNI2ED YOUTH EMPLOYMENT COMPETENCIES 

A. General Description of Youth Employment Competency Skill 
Areas 

* Pre-employment skills include world of work awareness, 
labor market knowledge, occupational information, values 
clarification and personal understanding, career 
planning and decision making, and job search techniques 
(resumes, interviews, applications, and follow-up 
letters). They also encompass survival/daily living 
skills such as using the phone, telling time, shopping, 
renting an apartment, opening a bank account, and using 
public transportation; and 

Work maturity skills include positive work habits, 
attitudes, and behavior such as punctuality, regular 
attendance, presenting a neat appearance, getting along 
and working well with others, exhibiting good conduct, 
following instructions and completing tasks, accepting 
constructive criticism from supervisors and co-workers, 
showing initiative and reliability, and assuming the 
responsibilities involved in maintaining a job. This 
category also entails developing motivation and 
adaptability, obtaining effective coping and problem- 
solving skills, and acquiring an improved self image. 

** Basic education skills include reading comprehension, 
math computation, writing, speaking, listening, problem 
solving, reasoning, and the capacity to use these skills 
in the workplace. 

* Job-specific skills — Primary job-specific skills 
encompass the proficiency to perform actual tasks and 
technical functions required by certain occupational 
fields at entry, intermediate or advanced levels. 
Secondary job-spejific skills entail familiarity with and 
use of set-up procedures, safety measures, work-related 
tenninology , recordkeeping and paperwork formats, tools, 
equipment and materials, and breakdown and clean-up 

rout ines. 
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B. Sufficiently Developed Systems for Youth Employment 
Competencies 

A sufficiently developed youth employment competency system must 
include the following structural and procedural elements: 

1 . Quantifiable Learning Objectives 

• PIC-recognized competency statements that are quantifiable, 
employment-related, measurable, verifiable learning 
objectives that specify the proficiency to be achieved as a 
result of program participation. 

Employment competencies/quantifiable learning objectives approved 
by the PIC as relevant to the SDA must include a description of 
the skills/knowledge/attitudes/behavior to be taught, the levels 
of achievement to be attained, and the means of measurement to be 
used to demonstrate competency accomplishment. The level of 
achievement selected should enhance the youth's employabili ty and 
opportunities for postprogram employment. 

2 . Related Curricula, Training Modules, and Approaches 

• Focused curricula, training modules, or behavior 
modification approaches which teach the employment 
competencies in which youth are found to be deficient. 

Such related activities, components, or courses must encompass 
participant orientation, work-site supervisor/instructor/ 
community volunteer training, and staff development endeavors as 
appropriate. They also must include, as appropriate, relevant 
agreements, manuals, implementation packages, instructions, and 
guidelines. A minimum duration of training must be specified 
which allows sufficient time for a youth to achieve those skills 
necessary to attain his/her learning objectives. 

3 . Pre-Assessment 

• Assessment of participant employment competency needs at 
the start of the program to determine if youth require 
assistance and are capable of benefitting from available 
services. 

A minimum level of need must be established before a participant 
is eligible to be tracked as a potential "attained PIC-recognized 
youth employment competency" outcome. All assessment techniques 
must be objective, unbiased and conform to widely accepted meas- 
urement criteria. Measurement methods used must contain clearly 
defined criteria, be field tested for utility, consistency, and 
accuracy, and provide for the training/preparation of all 
raters/scorers . 
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4. Post-Assessment (Evaluation) 

• Evaluation of participant achievement at the end of the 
program to determine if competency-based learning gains 
took place during project enrollment. 

Intermediate checking to track progress is encouraged. All 
evaluation techniques must be objective, unbiased and conform to 
widely accepted evaluation criteria. Measurement methods used 
must contain clearly defined criteria, be field tested for 
utility, consistency, and accuracy, and provide for the 
training/preparation of all raters/scorers. 

5. Employability Development Planning 

• Use of assessment results in assigning a youth to 
appropriate learning activities/sites in the proper 
sequence to promote participant growth and development, 
remedy identified deficiencies, and build upon strengths. 

6. Documentation 

Maintenance of participant records and necessary reporting 
of competency-based outcomes to document intra-program 
learning gains achieved by youth. 

7. Certification 



Proof of youth employment competency attainment in the form 
of a certificate for participants who achieve predetermined 
levels of proficiency to use as evidence of this 
accomplishment and to assist them in entering the labor 
market. 
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C. Guidelines for Ensuring Consistency in the Reporting of 
Pre-Employment/Work Maturity Skill Competencies 

Individuals should demonstrate proficiency in each of the follow- 
ing 11 core competencies. In order for an attainment to be 
reported in the area of pre-employment/wor k maturity, at least 
one PIC-certi f ied competency statement must be developed/ 
quantified in each of the following 11 core competencies — 
provided that at least 5 of these learning objectives were 
achieved during program intervention: 

1. Making Career Decisions 

2 . Using Labor Market Information 

3 . Preparing Resumes 

4 . Filling Out Applications 

5 . Interviewin g, 

6. Being Consistently Punctual 

7. Maintaining Regular Attendance 

8 . Demonstrating Positive Attitudes/Behavior 

9 . Presenting Appropriate Appearance 
Exhibiting Good Interpersonal Relations 
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li. Completing Tasks Effectively 




Appendix C 



Definitions of Terms Necessary 
for Completion of Reports 



EMPLOYMENT/TRAINING SERVICES 

ASSESSMENT — services are designed to initially determine 
each participant's employability , aptitudes, abilities and 
interests, through interviews, testing and counseling to 
achieve the applicant's employment related goals. 

FOLLOW-UP — is the collection of information on a terminee's 
employment situation at a specified period after termination 
from the program, 

INTAKE — includes the screening of an applicant for eligi- 
bility and: (1) a determination of whether the program can 
benefit the individual; (2) an identification of the 
employment and training activities and services which would 
be appropriate for that individual; (3) a determination of 
the availability of an appropriate employment and training 
activity; (4) a decision on selection for participation and 
(5) the dissemination of information on the program. 

OUTREACH activity involves the collection, publication and 
dissemination of information on program services directed 
toward economically disadvantaged and other individuals 
eligible to receive JTPA training and support services. 

YOUTH EMPLOYABILITY ENHANCEMENT TERMINATION 

An outcome for youth, other than entered unsubsidized 
employment, which is recognized as enhancing long-term 
employability and contributing to the potential for long-term 
increase in earnings and employment. The five youth 
employability enhancement outcomes are; 

(1) Demonstrated proficiency in youth employment 
competencies as defined by the PIC in two or more of the 
following three skill areas in which the terminee was 
deficient at enrollment: pre-employment/work maturity, 
basic education, or job-specific skills. During PT '88 
defiSltiSX ^" ^^^^^ area will aeet the 

(2) Entered an occupational-skills employment/training 
program, not funded under Title ll of the JTPA, which 
builds upon and does n ot duplicate training received under 
Ti t le II , 

(3) Returned to full-time school if, at time of intake, 
the participant was not attending school and had not 
obtained a high schooT~d iploma or equivalent, 
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(4) Completed, during enrollment, a level of educational 
achievement which had not been reached at entry. Levels of 
educational attainment are elementary, secondary, and 
post-secondary. Completion standards shall be governed by 
State standards or, at the Governor's discretion, local 
standards at the elementary level; shall include a high 
school diploma, GED Certificate or equivalent at the 
secondary level, and shall require a diploma or other 
written certification of completion at the post-secondary 
level. NOTE: To obtain credit, completion of a major 
level of education must result primarily from participation 
in a JTPA activity. 

(5) Completed program objectives as defined in approved 
exemplary youth project plans if, at time of entry, the 
participant was 14 or 15 years of age. 

EDUCATION STATUS 

SCHOOL DROPOUT — An adult or youth (aged 14-21) who is not 
attending school full-time and has not received a high school 
diploma or a GED certificate. 

STUDENT — An adult or youth (aged 14-21) who has not 
received a high school diploma or GED certificate and is 
enrolled full-time in an elementary, secondary or 
postsecondary-level vocational, technical, or academic 
school or is between school terms and intends to return to 
school . 

HIGH SCHOOL GRADUATE OR EQUIVALENT (No Post-High School) — 
An adult or youth (aged 14-21) who has received a high school 
diploma or GED Certificate, but who has not attended any 
postsecondary vocational, technical, or academic school. 

POST HIGH SCHOOL ATTENDEE — An adult or youth (aged 14-21) 
who has received a high school diploma or GED certificate 
and has attended (or is attending) any postsecondary-level 
vocationax, technical, or academic school. 

FAMILY STATUS 

SINGLE HEAD OF HOUSEHOLD — A single, abandoned, separated, 
divorced or widowed individual who has responsibility for one 
or more dependent children under age 18. 

RACE/ETHNIC GROUP 

WHITE (NOT HISPANIC) — A person having origins in any of the 
original peoples of Europe, North Africa, or the Middle East. 

BLACK (NOT HISPANIC) — A person having origins in any of the 
black racial groups of Africa. 
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HISPANIC — A person of Mexican, Puerto Rican, Cuban, Central 
or South American, or other Spanish culture or origin 
(including Spain), regardless of race. NOTE : Among persons 
from Central and South American countries, only those who are 
of Spanish origin, descent, or culture should be included in 
the Hispanic category. Persons from Brazil, Guiana, and 
Trinidad, for example, would be classified according to their 
race, and would not necessarily be included in the Hispanic 
category. Also, the Portugese should be excluded from the 
Hispanic category and should be classified according to their 
race . 

AMERICAN INDIAN OR ALASKAN NATIVE — A person having origins 
in any of the original peoples of North America, and who 
maintains cultural identification through tribal affiliation 
or community recognition. 

ASIAN OR PACIFIC ISLANDER — A person having origins in any 
of the original peoples of the Far East, Southeast Asia, the 
Indian subcontinent (e.g., India, Pakistan, Bangladesh, Sri 
Lanka, Nepal, Sikkim, and Bhutan), or the Pacific Islands. 
This area includes, for example, China, Japan, Korea, the 
Philippine Islands, ar.J Samoa. Hawaiian natives are to be 
recorded as Asian or Pacific Islanders. 

OTHER BARRIERS TO EMPLOYMENT 

LIMITED ENGLISH LANGUAGE PROFICIENCY — Inability of an 
applicant, whose native language is not English, to 
communicate in English, resulting in a job handicap. 

HANDICAPPED INDIVIDUAL — Refer to Sec. 4(10) of the Act. 
Any individual who has a physical or mental disability which 
for such individual constitutes or results in a substantial 
handicap to employment. NOTE: This definition will be used 
for performance standards purposes, but is not required to be 
used for program eligibility determination (Sec. 4(8) (E)). 

OFFENDER — For reporting purposes, the term "offender" is 
defined as any adult or youth who requires assistance in 
overcoming barriers to employment resulting from a record of 
arrest or conviction (excluding misdemeanors) . 

READING SKILLS BELOW 7th GRADE LEVEL — An adult or youth 
assessed as having English (except in Puerto Rico) reading 
skills below the 7th grade level on a generally accepted 
standardized test. 

NOTE: The following other methods of determination may be 
used : 

° A school record of reading level determined within the 
last 12 months. 

Rev, 9/88 
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** If an applicant is unable to read and therefore cannot 
complete a self -application for the JTPA/EDWAA program, 
s/he may be considered to have English reading skills 
below the 7th-grade level. 

® Individuals with any of the following may be considered 
to have English reading skills above the 7th-grade 
level : 

- A GED certificate received within the last year. 

- A degree (usually a BA or BS ) conferred by a 4-year 
college, university or professional school. 

If there is any question regarding reading ability, a 
standardized test should be administered. 

LONG-TERM AFDC RECIPIENT — An adult or youth listed on the 
welfare grant who had received cash payments under AFDC (SSA 
Title IV) for any 24 or more of the 30 months prior to JTPA 
eligibility determination and who was a welfare recipient (as 
defined below) at the time of such determination. 

U.C. STATUS 

UNEMPLOYMENT COMPENSATION CLAIMANT — Any individual who has 
filed a claim and has been determined monetarily eligible for 
benefit payments under one or more State or Federal 
unemployment compensation programs, and who has not exhausted 
benefit rights or whose benefit year has not ended. 

LABOR FORCE STATUS 

EMPLOYED — (a) An individual who, during the 7 consecutive 
days prior to application to a JTPA program, did any work at 
all: (i) as a paid employee; (ii) in his or her own 
business, profession or farm, or (iii) worked 15 hours or 
more as an unpaid worker in an enterprise operated by a 
member of the family? or (b) an individual who was not 
working, but has a job or business from which he or she was 
temporarily absent because of illness, bad weather, vacation, 
labor-management dispute, or personal reasons, whether or not 
paid by the employer for time off, and whether or not seeking 
another job. (This term includes members of the Armed Forces 
on active duty, who have not been discharged or separated; 
participants in registered apprenticeship programs; and 
self-employed individuals.) 

EMPLOYED PART-TIME ~ An individual who is regularly 
scheduled for work less than 30 hours per week. 
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UNEMPLOYED — An individual who did not work during the 7 
consecutive days prior to application for a JTPA program, who 
made specific efforts to find a job within the past 4 weeks 
prior to application, and who was available for work during 
the 7 consecutive days prior to application (except for 
temporary illness) . 

UNEMPLOYED: 15 OR MORE WEEKS OF PRIOR 26 WEEKS — An 
individual who is unemployed (refer to definition above) at 
the time of eligibility determination and has been unemployed 
for any 15 or more or the 26 weeks immediately prior to such 
determination, has made specific efforts to find a job 
throughout the period of unemployment, and is not classified 
as "Not in Labor Force". 

NOT IN LABOR FORCE — A civilian 14 years of age or over who 
did not work during the 7 consecutive days prior to 
application for a JTPA program and is not classified as 
employed or unemployed. 

WELFARE GRANT INFORMATION 

WELFARE RECIPIENT — An individual listed on the welfare 
grant who was receiving cash payments under AFDC (SSA Title 
IV), General Assistance (State or local government), or the 
Refugee Assistance Act of 1980 (PL 96-212) at the time of 
JTPA eligibility determination. For reporting and 
performance standards purposes, exclude those individuals who 
receive only SSI (SSA Title XVI)"! 

PROGRAM COSTS 

ACCRUED EXPENDITURES — The allowable charges incurred during 
the program year to date requiring provision of funds for: 
(1) goods and other tangible property received; and (2) costs 
of services performed by employees, contractors, 
subrecipients, and other payees. 

NOTE: These charges do not include "resources on order". 
I.e., amounts for contracts, purchase orders and other 
obligations for which goods and/or services have not been 
received . 
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WORKER ADJUSTMENT PROGRAM ANNUAL PROGRAM REPOFT (ETA 9019) 

1. Purpose . The Economic Dislocation and Worker Adjustment 
Assistance Act (EDWAA) Annual Program Report (WAPR) displays 
cumulative data on participation, termination, performance 
measures and the socio-economic characteristics of all 
terminees on an annual basis. The information will be used 
to determine levels of program service and performance 
measures. Selected information will be aggregated to provide 
quantitative program accomplishments on a local. State, and 
national basis. 

2. General Instructions . The Governor will submit: (1) a 
separate WAPR for each designated Substate Area (SSA) ; (2) a 
separate Statewide WAPR summary of the SSA report data (this 
Statewide WAPR summary of individual SSA data should not be 
submitted by single-SSA States); (3) a separate WAPR covering 
participants and terminees in statewide, regional or 
industrywide projects funded under Section 302(c)(1)(B) of 
the Act; and (4) a separate WAPR covering participants and 
terminees in projects funded under Secretary's National 
Reserve Grants (Section 302(a)(2)). Recipients may determine 
whether the reports are submitted on WAPR forms or as a 
computer printout, with data, including signature and title, 
date signed and telephone number, arrayed as indicated on the 
WAPR form. If revisions are made to the WAPR data after the 
reporting deadline, revised copies of the WAPR should be 
submitted to DOL as soon as possible according to the 
required reporting procedures. Submittal of one or more 
WAPRs with revised information for Total Participants, Total 
Terminations and/or Total Program Costs (Federal Funds) 
usually will require submittal of a revised WQFR which 
includes the final quarter of the same program year. 

NOTE: For WAPR reporting purposes, EDWAA shall refer 
to: (1) programs operated by Substate grantees with 
funds authorized under Sections 302(c)(2) and 302(d) or 
otherwise distributed by the Governor under Section 
302(c)(1)(E) and (2) projects operated by the Governor 
with funds authorized under Sections 302(c)(1) and 
302(a) (2) of the Act. 

The reporting period begins on the starting date of each Job 
Training Partnership Act (JTPA) program year, as stated in 
Section 161 of the JTPA. Reports are due in the National and 
Regional Offices no later than 45 days after the end of each 
program year. Two copies of the WAPR are to be provided to: 

U.S. Department of Labor, ETA 
ATTN: TSVR — Room S-5306 
200 Constitution Avenue, N.W. 
Washington, D. C. 20210 

(Rev. 2/90) 



At the same time an additional copy of the WAPR is to be 
provided to the appropriate Regional Administrator for 
Employment and Training in the DOL Regional Office that 
includes the State in which the JTPA recipient is located. 

NOTE: The current JASR, ETA 8580 (June 1988), is to continue 
to be used for programs operating solely with PY88 and 
earlier year's Title III funds. The WAPR is to be used 
beginning July 1, 1989, for °Y89 programs funded under EDWAA 
(including carryin funds used for EDWAA purposes, if any). 

Facsimile of Form . See the following page. 

Instructions for Completing the Worker Adjustment Program 
Annual Program Report (WAPR) . 

a. State/Substate Area Name and Address 

On separate Section 302(c)(1) and Section 302(a)(2) 
reports, enter the name and address of the State agency 
that will administer the Statewide programs. For SSA 
reports, enter the name and address of the Substate 
grantee that will administer the sSA programs. 

b. Report Type 

Designate the type of report data provided on this WAPR. 
If this is an SSA report, also enter the ETA-assigned 
Substate Area Code number. 

c . Report Period 

Enter in "From" the beginning date of the designated JTPA 
program year and enter in "To" the ending date of that 
program year . 

d. Signature and Title (at bottom of the page) 

The authorized official signs here and enters his/her 
title. 

e. Date signed 

Enter the date the report was signed by the authorized 
official. 

f . Telephone Number 

Enter the area code and telephone number of the 
authorized official. 
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Annual Program Report (WAPR; 



U.S. Depa'tnen: o* ^£Dor 

Empioyme'^: ana Training Aominisirauon 



a. suitt/substite Area Name and Address 





b. ^apoHType 

( ) SSA # 

( ) GovStaiawida 
( ) SacyN/Rasv 

c. Repon Period 



0MB No. 120M274 
Expires 12/31/92 



From: 7/1/19 



To: 6/30/19 



I. PARTICIPATION AND TERMINATION SUMMARY 



A. ittued Certificate of Continuing Eligibility (CCE) 



Corwurrent 
ParticiDants 



B. 



All 

Participants 



B. TOTAL PARTICIPANTS 



1. All CCEs Reaeenr)ed fo r Retraining 
C. TOTAL TERMINATIONS 



1. Entered Unsubsidized Enfipioyment From Retraining 



a. Relocated Out of Area 



2. Entered Unsubttdized Employment From Basic Readjustment Services ONLY 



3. Called BacK/Rematned with the Layoff Employer 



4. Transferred to Other JTPA Programs 



5. Entered Non.J7PA Training 



6, All Other Terminations 
1 1| II. TERMINEE CHARACTERISTICS AND PERFORMANCE MEASURES SUMMARY 




1^1 S<nple Heed of Houethold With DepefH<ent(i) Under Ape 18 



12 
13 
14 
15 
16 
17 



Black (Not Hispanic) 



Hispanic 



Amer teen indiar. or Alas kan Native 
Asian Of Pacific lelander 
Limlfd English LanQuape Proffcitncy 

Handicapped 

Riadino Skills Bek>w 7th4kade Level 




BEST COPY AVAILABi 



Report Period 



From 
7/1/19 



To 

6/X/19 




II. TERMINEE CHARACTERISTICS AND PERFORMANCE luiEASURES SUMMARY - CONTINUED 



All Tflrmir)tet 



20 


1)1 


U.C Clalnr\ant 




21 




Unemployed 15 or More Weeks of Prior 26 Weeks 


22 
23 


ss 


Veteran (Total) 


VlatnarrwEra i 




24 


Average Weeks Panicipated 




25 


Average Hourly Wage - Pre-Program 


! 


26 


1 Average Hourly Wage at Termir^ation j 


27 


1 Total Program Costs (Federal Funds) 


26 1 N«eds*Raiat0d Piymenu 




29 Total Availabit Faderai Funds 




111. FOLLOW-UP INFOI^MATION 


X 


Employment Rata at Follow-up | 


31 


Average Hourly Wage at Follow-up j 


32 


Average Number of Weeks Worked In Follow-up Period j 


33 


Sample Size | 


34 


^sponse Rata 






IV. RETRAININQ/BA8IC READJUSTMENT SERVICES 


35 


Received Basic Readjustment Services ONLY 




36 


Received ANY Retraining Activity 




37 


Uss than 26 Weeks 






36 


26 or More Weeks 






39 


Connpiettd Daearoom Training: Basic Education or Attained QED 




40 


Completed Orvth»-Job Training 




41 


Compltttd Othr Occupational Skills Training 






Pagt 2 of 2 Pagt* 



(March 1868) 
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^' General Inf orma tioii . Unless otherwise indicated/ data 

reported on characteristics of terminees should be based on 
information collected at the time of eligibility 
determination , 



CHARACTERISTICS INFORMATION OBTAINED ON AN INDIVIDUAL AT THE TIME 
OF ELIGIBILITY DETERMINATION FOR THE RECIPIENT'S EDWAA PROGRAM 
SHOULD NOT BE UPDATED WHRN THE INDIVIDUAL TERMINATES FROM THE 
EDWAA PROGRAM. 




NOTE: Recipients shall ensure that individuals are enrolled 
within 45 days of the date of application or a new application 
must be taki n (20 CFR 629.1). This 45-day period for Certificate 
of Continuing Elibibility (CCE) holders should begin upon 
completion of a full participant record, usually at the time of 
CCE redemption, as this may be at a location other than where the 
CCE was issued and such information taken earlier to determine 
eligibility may not be readily available at the redemption site. 

SECTION I - PARTICIPATION AN D TERMINATION SUMMARY 

Section I displays the EDWAA program's accomplishments in terms 
of the total cumulative number of participants in the program, 
the number and types of terminations from the program, and ti.e 
number of CCEs issued/redeemed, as of the end of the reporting 
period. ^ ^ 

In Section I, Column A., Item I.B., enter individuals who are 
concurrent participants and are receiving Basic Readjustment 
Services and/or Retraining under another EDWAA-funded 
grantee/program, JTPA title or have entered non-JTPA/EDWAA 
training for the completion of the initially determined training 
objective, prior to termination from their initial EDWAA 
program. Column A. is a sub-breakout of Column B. for this line 
1 tem • 



In Section I, Column A., Item I.e., enter terminees who have been 
concurrent participants and have received Basic Readjustment 
Services and/or Retraining under another EDWAA-funded 
' grantee/program, JTPA title or have received non-JTPA/EDWAA 
training for the completion of the initially determined training 
objective, prior to termination from their initial EDWAA 
program. Also, distribute these terminees between Items I.C.I, 
and I.e. 2., as appropriate, on the basis of the final type'of 
termination from the EDWAA program of final participation. 
Column A. is a sub-breakout of Column B. for these line items. 
NOTE: An individual included in a line item entry in Column A 
also must be included in the entry for the same line item in 
Column B. 



Entries for Items I. A, I.B. and I.e. are cumulative from the 
beginning of the program year through the end of the reporting 
period. 
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Item I. A. Issued Certificate of Continuing Eligibility (CCE) 

Enter the total number of applicants to whom a Certificate of 
Continuing Eligibility was issued during this program year, as 
provided in Section 316(b) of the Act, by the Substate grantee 
through the end of the reporting period. NOTE: Enter zero for 
this item on the Governor's Statewide report (Section 302(c)(1)). 

Item I.B. Total Participants 

Enter by column the total number of participants who are or were 
receiving employment, training or services (except post-termina- 
tion services) through the end of the reporting period, including 
both those on board at the beginning of the designated program 
year and those who have entered during the program year. 

"Participant" means any individual who has: (1) been determined 
eligible for participation upon intake; and (2) started receiving 
employment, training, or services (except post-termination 
services) funded under the Act, following intake. Individuals 
who receive only outreach and/or intake and initial assessment 
services or postprogram follow-up are excluded. 

NOTE: Also exclude individuals who receive only Rapid Response 
Assistance and information, per Section 314(b), provided by the 
State's Dislocated Worker Unit. 

If individuals receive concurrent employment, training and/or 
services under more than one title/program, they are to be 
considered participants in both titles/programs for purposes of 
recording actual number of weeks participated, dollars expended, 
and other pertinent data. Individuals who initially participate 
in EDWAA funded activity and subsequently participate in any 
other EDWAA (or non-EDWAA) funded activity, FOR THE COMPLETION OF 
THE INITIALLY DETERMINED TRAINING OBJECTIVE, may be considered to 
be concurrent participants in each program. 

The sum of the entries (all SSAs in a State) in Item I.E., Total 
Participants, and in Item I.C., Total Terminations, of the WAPR 
should equal the entries for Substate Grantees in Column B., 
Lines 16 and 17, respectively, of the WQFR for the final quarter 
of the same program year for the same recipient. 

The entries in Item I.B. and in Item I.C. of the Governor's 
Statewide WAPR should equal the entries for Governor's Reserve in 
Column A., Lines 16 and 17, respectively, of the WQFR for the 
final quarter of the same program year for the same recipient. 

The entries in Item I.B. and in Item I.C. of the Secretary's 
National Reserve WAPR should equal the entries for the 
Secretary's National Reserve in Column C, Lines 16 and 17, 
respectively of the WQFR, for the final quarter of the same 
program year for the same recipient. 
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Item I.B.l. All CCES Redeemed for Retraining 

Enter the total number of unexpired participant CCEs, regardless 
of year issued, redeemed for Retraining during this program year 
by the Substate grantee, through the end of the reporting 
period. Include all CCEs so redeemed that were issued by any 
Substate grantee for periods not to exceed 104 weeks prior to 
redemption. This item is a sub-breakout of Item I.B. 

Item I.e. Total Terminations 

Enter by column the total number of participants who terminated 
(as defined below) from the program during the reporting period. 
Include all participants who received no Basic Readjustment 
Services (except supportive services and/or counseling) or 
Retraining for 90 days. This iten is the sua of I tens I.C.I, 
through I.C.6 . — — . 

"Termination" means the separation of a participant from the pro- 
gram who is no longer receiving Basic Readjustment Services or 
Retraining under EDWAA. Individuals may be considered 
participants for up to 90 days after last receipt of Basic 
Readjustment Services or Retraining, during which time they may 
continue to receive supportive services, as provided for in 
Section 314 (c)(15) and Section 4(24) of the^Act. 

For purposes of calculating average weeks participated, this 
single period of up to 90 days between "last receipt of Basic 
Readjustment Services or Retraining under EDWAA" and actual date 
of termination is defined as "inactive status" and is not to be 
included in "Average Weeks Participated". Terminees may continue 
to receive counseling necessary to assist in the retention of 
employment, for not more than 6 months following last receipt of 
Basic Readjustment Services (Section 314 (c)) or Retraining 
(Section 314 (d) ) . 

Participants who have transferred from one title to another, or 
between programs of the same title, should be recorded as 
terminations from the title or program of initial participation 
and included as participants in the title or program into which 
they have transferred. If they are concurrent participants in 
more than one title or program, the type of termination 
determined for the fina) program should be recorded for all 
programs for these participants. 

For purposes of calculating average weeks participated for such 
concurrent EDWAA program participants, the period between "last 
receipt of Basic Readjustment Services and Retraining funded 
under a given EDWAA program" (i.e., SSA, Governor's Reserve or 
Secretary's National Reserve) and actual date of termination from 
that EDWAA program is defined as "inactive status" and is not to 
be included in Line 24. 
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Item I.C.I. Entered Unsubsidized Employment from Retraining 

Enter by column the total number of participants who, at 
termination, entered full- or part-time unsubsidized employment 
from Retraining through the end of the reporting period. (These 
participants may or may not have received Basic Readjustment 
Services.) This item is a sub-breakout of Item I.e. 

NOTE: Unsubsidized employment means employment not financed from 
funds provided under the Act and includes, for JTPA/EDWAA 
reporting purposes, entry into the Armed Forces of the U. S., 
entry into employment in a registered apprenticeship program and 
terminees who become self-employed. 

Item I.C.I. a. Relocated Out of Area 

Enter the total number of participants who, at termination from 
other Retraining, entered unsubsidized employment after receiving 
relocation assistance and relocating outside the Substate Area 
which provided such relocation assistance, or within or outside 
of the State, if this assistance was provided by a Statewide SSA 
or by a program administered by the Governor. This item is a 
sub-breakout of Item l.C.l. 

Item I.e. 2. Entered Unsubsidized Employment From Basic 
Readjustment Services ONLY 

Enter by column the total number of participants who, at 
termination, entered full- or part-time unsubsidized employment 
from Basic Readjustment Services ONLY through the end of the 
reporting period. This item is a sub-breakout of Item l.C. 

Item I.e. 3. ealled Back/Remained With the Layoff Employer 

Enter the number of terminees from the EDWAA program who, after 
being laid off by an employer, were recalled by that employer to 
a permanent job at the same or another location. Also include 
EDWAA program termineea who remained in a permanent job with an 
employer after receipt of a layoff notice from that employer. 
This item is a sub-breakout of Item I.e. NOTE: Do not include 
such terminees in the entry for Items I.e.l. or I.e. 2., above. 

Item I.e. 4. Transferred to Other JTPA Programs 

Enter the number of terminees who transferred to and entered 
programs funded under another JTPA title including Title 
Ill-Formula and Title Ill-National Reserve. Also include on this 
line terminees vjuo transferred to and entered EDWAA programs 
operated by another Substate grantee, or who transferred to and 
entered EDWAA programs operated by the Governor (and 
conversely) . This item is a sub-breakout of item l.C. 

(Rev. 2/90) 
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Item I.e. 5. Entered Non-JTPA Training 

Enter the number of terminees who entered, during the program 
year, training not funded with JTPA monies. This item is a 
sub-breakout of Item I.C. 

Item I.C. 6. All Other Terminations 



Enter by column the total number of participants who were termi- 
nated for reasons other than those in Items I.C.I, through 
I.C. 5., successful or otherwise, through the end of the reporting 
per iod . 

SECTION II - TERMINEE PERFORMANCE MEASURES INFORMATION 

Section II displays performance measures/parameters information. 
As indicated previously, data reported on characteristics of 
terminees should be based on information collected at time of 
eligibility determination unless otherwise indicated. 

Governors may develop any participant record which meets the 
requirements of Section 629.35(c) and (d) of the JTPA 
regulations. The DOL/ETA Technical Assistance Guide ; The JTPA 
Participant Record , dated May 1983, may be used as a refeTence . 

Line Item Definitions and Instructions 

Sex 

Line 1 Male 
Line 2 Female 

Distribute the terminees according to Sex. The sum of Lines 1 
and 2 should equal Item I.C. 



Age 

Line 3 29 and Under 

Line 4 30-44 

Line 5 45-54 

Line 6 55 and Over 



Distribute the terminees according to Age. The sum of Lines 3 
through 6 should equal Item I.C. 

Education Status 



Line 7 
Line 8 
Line 9 
Line 10 



Less Than High School 

High School Graduate or Equivalent (No Post-High School) 
Post-High School At .indee 

College Graduate and Above 



Distribute the terminees according to Education Status. The sum 
of Lines 7 through 9 should equal Item I.C. NOTE: Line 10 is a 
sub-breakout for a specific group included in Line 9. 
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Family Sta tus 

Line 11 ^^"5^® °^ Household with Dependent(s) Under Age 

Enter the total number of terminees for whom the above Family 
Status classification applies. 

Race /E thnic Gro up 

Line 12 White (Not Hispanic) 
Line 13 Black (Not Hispanic) 
Line 14 Hispanic 

Line 15 American Indian or Alaskan Native 
Line 16 Asian or Pacific Islander 

Distribute the terminees according to the Race/Ethnic Groups 
listed above. For purposes of this report, Hawaiian Natives are 

12 throuoT?^ ^ "'""'^^^ Islander". The sum of Lines 

12 through 16 should equal Item I.e. 

Other Ba rriers to E mployment 

Line 17 Limited English Language Proficiency 
Line 18 Handicapped 

Line 19 Reading Skills Below 7th Grade Level 

o?^!r n^^ total number of terminees for whom each of the above 
Other Barriors to Employment apply. 

§ii2i£ its_S t at u s 

Line 20 U. c. (Unemployment Compensation) Claimant 

hr!!^5^^ "If"^^'' °^ terminees for whom each of the above 

benefits status classifications apply. 

L abor Force Status 

Line 21 Unemployed: 15 or More Weeks of Prior 26 Weeks 

po^^I °^ terminees for whom the above Labor 

Force Status classification applies. 

Vete r an Status 

Line 22 Veteran (Total) 
Line 23 Vietnam Era 

Enter the total number of terminees for whom each of the above 
ll^^'in'l ^J^ff^fications apply, as defined in siTtion 4 (26) (A) (B) 
and (D of the Act. Line 23 i.' a sub-breakout for a specif c 
group included in Line 22. oFe^ii-ic 
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Qtjlgg_ Prog rani Information 

Line 24 Average Weeks Participated 

Enter the average number of weeks of participation in the EDWAA 
program for all terminees. Weeks of participation include the 
period from the date an individual becomes a participant in EDWAA 
through the date of a participant's last receipt of Basic 
Readjustment and/or Retraining. Exclude the single period of up 
to 90 days during which an individual may remain in an inactive 
status prior to termination. Time in inactive status for all 
terminees should not be counted toward the actual number of weeks 
participated. Inactive status is defined as that period between 
"last receipt of Basic Readjustment Services and/or Retraining 
under EDWAA" and actual date of termination. (See Item I.e.) 

To calculate this entry: Count the number of days participated 
for each terminee/ including weekends, from the start date of 
his/her participation in EDWAA until his/her last receipt of 
Basic Readjustment Services and/or Retraining under EDWAA. 
Divide this result by 7. This will give the number of weeks 
participated for that terminee. Sum all the terminees' weeks of 
participation and divide the result by the number of terminees, 
as entered in Item I.e. This entry should be reported to the 
nearest whole week. 



Line 25 Average Hourly Wage — Pre-Program 

Enter the average hourly pre-EDWAA wage of all terminees. In 
calculating this average, use the hourly wage from the job of 
dislocation. Those terminees who had no pre-EDWAA employment 
should be counted as "$0.00" hourly wage. 

To calculate this entry: Sum the pre-program hourly wage for all 
terminees shown in Item i.e. Divide the result by the number of 
terminees shown in item i.e. NOTE: For the calculation, use 
the hourly wage regardless of whether the individual was employed 
fu^l- or part-time. ' 

Line 26 Average Hourly Wage at Termination 

Enter the average hourly wage at termination for the total number 
of terminees in Items I.C.l. through I.e. 3. 

To calculate this entry: Sum the hourly wage at termination for 
all the terminees shown in Items I.e.l. through I.e. 3. Divide 
the result by the number of terminees shown in Items I e 1 
through I.e. 3. 

Hourly wage includes any bonuses, tips, gratuities and 
coDDissions earned. 
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Line 27 Total Program Costs (Federal Funds) 

Enter the total accrued expenditures, through the end of the 
reporting period, of the funds allocated to the SSA under 
Sections 302(c)(2) and 302(d) of the Act or otherwise distributed 
by the Governor to the SSAs under Section 302(c)(1)(E). On the 
separate WAPRs : (1) covering participants and terminees projects 
funded under Section 302(c)(1); (2) or participants and terminees 
in Secretary's National Reserve Grants projects funded under 
Section 302(a)(2) of the Act, enter the total expendidures for 
all participants and terminees served in such programs through 
the end of the reporting period. Include, as appropriate, 
accrued expenditures against JTPA Title III funds provided for 
PY88 and PY87 which were carried into EDWAA and used for EDWAA 
purposes, if any. Include expenditures of Federal funds only. 

NOTE: Entries will be nade to the nearest dollar. Negative 
entries are not acceptable. The WAPR program cost data will be 
compiled on an accrual basis. If the recipient's accounting 
records are not normally maintained on an accrual basis, the 
accrual information should be developed through an analysis of 
the records on hand or on the basis of best estimates. 

The sum of the entries for Line 27, Total Program Costs, of the 
WAPR (i.e., total for the State's SSAs under EDWAA) should equal 
the entry for Column B., Line 10, Total Accrued Expenditures 
(Substate Grantees) of the WQFR for the same recipient that 
includes the final quarter of the same program year. 

Line 27 of the Statewide WAPR for the Governor's Reserve activity 
(Section 302(c)(1)) should equal Column A., Line 10 of the WQFR 
for the same recipient that includes the final quarter of the 
same program year; and Line 27 on the WAPR for the Secretary's 
National Reserve Grants (Section 302(a)(2)) to the state should 
equal Column B., Line 5 on that WQFR. 

Line 28 Needs-Related Payments 

Enter the total accrued expenditures for needs-related payments 
to eligible dislocated workers who do not qualify or have ceased 
to qualify for Unemployment Compensation, in order to enable such 
workers to participate in training or education programs under 
EDWAA (Section 314(e)). This is a sub-breakout of Line 27. 

Line 29 Total Available Federal Funds 

Enter the total Federal funds available for the EDWAA program 
described on this report including (1) unexpended funds carried 
over from previous program years, (2) funds allocated or awarded 
for this program year, and (3) any reallocation that increased or 
decreased the amount of funds available for expenditure through 
the end of this reporting period. Entries will be made to the 
nearest dollar . 
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The sum of the entries for Line 29, Total Available Federal 
Funds, of the WAPR (i.e., total for the State's SSAS under EDWAA) 
should equal the entry for Column B., Line 9, Total Federal Funds 
Available (Substate Grantees) of the WQFR for the same recipient 
that includes the final quarter of the same program year. 

Line 29 of the Statewide WAPR for the Governor's Reserve activity 
(Section 302(c)(1)) should equal Column A., Line 9 of the WQFR 
for the same recipient that includes the final quarter of the 
same program year; and Line 29 on the WAPR for the Secretary's 
National Reserve Grants (Section 302(a)(2)) to the State should 
equal Column B., Line 1 on that WQFR. 

SECTION III - FOLLOW-UP INFORMATION 

Section III displays information based on follow-up data which 
must be collected through participant contact to determine an 

^°^?^ ^^^^""^ earnings, if any, during the 
13th full calendar week after termination and the number of weeks 
s/he was employed during the 13-week period. Follow-up data 
should be collected from participants whose 13th full calendar 
week after termination ends during the program year (the 
follow-up group). Thus, follow-up will be conducted for 
individuals who terminate during the first three quarters of the 
program year and the last quarter of the previous program year. 

For PY89 follow-up may be conducted for individuals who terminate 
during the first three quarters of the program year and 
postprogram data collection need not begin until October 1, 1989. 

In order to ensure consistency of data collection and to 
guarantee the quality of the follow-up information, follow-up 
procedures must satisfy certain criteria. (See the Follow-up 
Guidelines included in these WAPR instructions. Appendix A.) 

Other procedures used to collect the follow-up data are at the 
discretion of the Governors. 

NOTE: Every precaution must be taken to prevent a "response 
bias which could arise becaure it may be easier to contact par- 
ticipants who were employed at termination than those who were 

entered employment at termination are 
more likely to be employed at follow-up. Special ptocedures have 
been developed by which sSAs and States can monitor response 
bias. If your response rates for those who were and were not 
employed at termination differ by more than 5 percentage points, 
the follow-up entries for the WAPR must be calculated using the 
worksheet for AdDUSting Follow-up Performance Measures" in the 
Follow-up Technical Assistance Guide. If the respon se rates 
differ by 5 percentage points or le 3, the foliowiSg instructions 
for completing Lines 30-34 may be t d . 

Line 30 Employment Rate at Follow-up 

Enter the employment rate at follow-up. 
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Calculate the employment rate by dividing the total number of 
respondents who were employed (full-time or part-time) during the 
13th full calendar week after termination by the total number of 
respondents (i.e., terminees who completed follow-up 
interviews). Then multiply the result by 100. This entry should 
be reported to the nearest one decimal (00.0). 

Line 31 Average Hourly Wage at Follow-up 

Enter the average hourly wage of those employed (full-time or 
part-time) at follow-up. 

To calculate this entry: Sum the hourly wage (and, if 
appropriate, add tips, overtime, bonuses, etc.) of each 
respondent employed at follow-up. Divide the sum of hourly wage 
for all respondents employed during the 13th full calendar week 
after termination by the number of respondents employed at the 
time of follow-up. Respondents not employed at follow-up are not 
included in this average. 

Include any wages, bonuses, tips, gratuities, commissions and 
overtime pay earned. 

Line 32 Average Number of Weeks Worked in Follow-up Period 

Enter the average number of weeks worked in follow-up period. 

To calculate the average number of weeks worked (full-time or 
part-time), divide the sum of the number of weeks worked during 
the 13 full calendar weeks after termination for all respondents 
who worked, by the total number of all respondents, whether or 
not they worked any time during this 13-week follow-up period. 
This entry should be reported to the nearest one decimal (00.0). 

Line 33 Sample Size 

Enter by column the size of the actual sample selected to be con- 
tacted for follow-up. 

NOTE: If oversampling was used, the sample size should include 
all those selected, not just the required minimum sample size. 
Those deceased or severely incapacitated to the point of being 
unable to respond at follow-up may be excluded from the sample 
Size. ^ 

Line 34 Response Rate 

Enter the overall response rate, i.e., the percentage of complete 
surveys obtained. v,uiiit.j.cte 

To calculate the overall response rate, divide the number of 
terminees with complete follow-up information by the total number 
of terminees included in the follow-up sample (Line 33) and 
multiply by 100. This entry should be reported to the nearest 
whole percent. 
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NOTE: Complete follow-up information consists of substantive 
answers to the required follow-up questions and may not include 
"don't know", "no answer" or "don't remember". 

SECTION IV - RETRAINING/BASIC READJUSTM^'^NT SERVICES 

Section IV displays information relevant to program activities 
funded under EDWAA. 

Line 35 Received Basic Readjustment Services ONLY 

Enter the total number of terminees, regardless of type of 
termination, who received Basic Readjustment Services ONLY, as 
indicated in Section 314(c) of the Act. The sum of Lines 35 and 
36 should equal Item I.C. NOTE: Individuals who receive only 
outreach and/or intake and initial assessnent services or Rapid 
Response assistance are not participants/terminees. 

Line 36 Received ANY Retraining Activity 
Line 37 Less than 26 Weeks 

Line 38 26 or More Weeks 

Enter the total number of terminees, regardless of type of 
termination, who received ANY Retraining activity included in 
Section 314(d) of the Act. Lines 37 and 38 are sub-breakouts of 
Line 36 and should be used to distribute terminees who received 
ANY Retraining activity by actual length of stay in all 
Retraining activities, whether or not such Retraining was 
completed. These terminees may or may not have received Basic 
Readjustment Services. The sum of Lines 37 and 38 should equal 
Line 36. 

Note: Terminees who have received retraining activity funded 
under a cooperative agreement with : (1) other JTPA monies 
(i.e. 3%, 8%, Title II etc.) or (2) other than JTPA funds may be 
counted in Lines 36-41, PROVIDED SOCH TRAINING WAS FOR THE 
COMPLETION OP THE INITIALLY DETERMINED TRAINING OBJECTIVE. 

Line 39 Completed Classroom Training: Basic Education or 

Attained GED 
Line 40 Completed On-the-Job Training 
Line 41 Completed Other Occupational Skills Training 

Enter the total number of terminees for which each of these 
Retraining completion/attainment classifications apply. A 
terminee should be included in all appropriate categories. 

NOTE: Basic Education in Line 39 includes remedial reading, 
writing, mathematics and/or English for non-English speakers. 
Attainment of a GED or a high school diploma upon completion of 
any training also should be included. (A terminee may be counted 
only once in this line item, as appropriate.) 
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Appendix A 



Follow-up Guidelines 



To ensure consistent data collection and as accurate information 
as possible, procedures used to obtain follow-up information must 
satisfy the following criteria: 

* Participant contact should be conducted by telephone or 
in person. Mail questionnaires may be used in those 
cases where an individual does not have a telephone or 
cannot be reached . 



Participant contact must occur as soon as possible after 
the 13th full calendar week after termination but no 
later than the 17th calendar week after termination. 

Data reported are to reflect the individual's labor force 
status and earnings during the 13th full calendar week 
after termination and the number of weeks s/he was 
employed throughout the 13-week period after termination. 

Interview questions developed by DOL (see following 
Exhibit) must be used to determine the follow-up 
information reported on the WAPR. Respondents must be 
told that responding is voluntary and that information 
provided by them will be kept confidential. Other 
questions may be included in the interview. Attitudinal 
questions may precede DOL questions, but questions 
related to employment and earnings must follow. 

Attempts must be made to contact al_l individuals unless 
terminee populations are large enough to use sampling. 

At least six attempts may need to be made to contact 
ri.ough individuals in the follow-up group to obtain the 
required response rate. 

For each SSA and for Statewide and National Reserve 
reports (WAPR) , minimum response rates of 70% are 
required for each of the following two groups •>£ 
dislocated workers: those who entered employment at 
termination and those who did not enter employment at 
termination. The response rate is calculated as the 
number of terminees with complete follow-up information 
divided by the total number of terminees included in the 
group eligible for follow~up. 
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EXHIBIT 

Minimum Postprograin Data Collection Questions 

I want to ask you about the week starting on Sunday, , 

and ending on Saturday, , which was (last 

week/two/three/four weeks ago) . 

1. Did you do any work for pay during that week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour in that week? 

Dollars per hour 

4. HOW much extra, if any, did you earn in that week from 
tips, overtime, bonuses, commissions, or any work you did 
on the side, before deductions? 

Dollars 



Now I want to ask you about the entire 13 weeks from Sunday, 
, to Saturday, . 

5. Including the week we just talked about, how many weeks 
did you work at all for pay during the 13-week period? 

Weeks [Go to end] 



ALTERNATIVE QUESTIONS 
If answered "NO" to Queotion 1: 

Now I want to ask you about the entire 13 weeks from Sunday, 
t to Saturday, . 

6. Did you do any work for pay during that 13-week period? 

Yes [Go to 7] 

No (Go to end] 

7. How many weeks did you do any work at all for pay during 
that 13-week period? 
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T erminee Pop u lat io ns for F ollow-up 

Each program (SSA> Governor's Reserve/ Secretary's National 
Reserve) in EDWAA is responsible for conducting a follow-up of 
all or a sample of parti-'ipants who have terminated from that 
program. 

The ^universe" of terminees for the follow-up includes all 
participants who terminated from a program. Those participants 
who may have been concurrent participants in more than one JTPA 
title/EDWAA program will be in the universe for each. 

When selecting a sample from the universe, each title/program 
will be treated separately, so that an individual who had been a 
concurrent participant might be selected in one sample but not 
another. This, however, does not preclude the possibility that 
the participant might be selected in more than one sample. In 
the event that a concurrent participant has been selected in more 
than one sample, the responses collected from a single interview 
may be shared among the different titles/programs to avoid the 
necessity of multiple interviews with the same individual. 

When an individual who has terminated from one title/program and 
subsequently become a participant in another program (i.e., not a 
concurrent participant) is selected in the sample for both 
titles/programs, separate interviews must be conducted. Further, 
if an individual is selcted in one sample and is a participant in 
another title/program at the time of the inverview, regular 
follow-up information should be determined and recorded. 

Sampling Proce d ures 

Where sampling is used to obtain participant contact information. 

It is necessary to have a system which ensures consistent random 

selection of sample participants from all terminees in the qrouo 

requiring follow-up. ^ 

No participant in the follow-up group may be arbitrarily 
excluded from the sample. 

Procedures used to select the sample must conform to 
generally accepted statistical practice, e.g., a table of 
random numbers or other random selection techniques must 
be used. 





The sample selected for contact must meet minimum sample 
size requirements indicated in Table 1. 

The use of sampling will depend on whether the terminee 
populations are large enough to provide estimates which meet 
minimum statistical standards. If the number of terminees for 
whom follow-up is required is less than 138, sampling cannot be 
used. In such cases attempts must be made to contact all the 
appropriate terminees. 
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Table 1 

MINIMUM SAMPLE SIZES FOR FOLLOW-UP 

Number of Terminees Minimum Sample Sampling 

in_Fql low-up Po pul ation_ size Percent ages 

1 - 137 All 100 

138 - 149 137 94 

150 - 159 143 92 

160 - 169 149 89 

170 - 179 154 87 

180 - 189 159 85 

190 - 199 164 84 

200 - 224 175 82 

225 - 249 185 78 

250 - 274 194 

275 - 299 202 71 

300 - 349 217 67 

350 - 399 229 62 

400 - 449 240 57 

450 - 499 250 53 

500 - 599 265 50 

600 - 749 282 44 

750 - 999 302 38 

1,000 - 1,499 325 30 

1/500 - 1,999 338 22 

2,000 - 2,999 352 17 

3,000 - 4,999 364 12 

5,000 or more 383 7,3 
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Mi n imum Sampl e Size s for F o 1 low- up 

To determine the minimum number of terminees to be included in 
the follow-up sample/ refer to Table 1 in the following 
instructions. Find the row in the left-hand column that contains 
the planned number of dislocated worker terminees. The required 
minimum sample size is given in the middle column of that row. 
The last column gives sampling percentages that will assure that 
the minimum sample size is obtained. 

C orrecting for Difference s in Responae_Ra tes 

Different response rates for those terminees who entered employ- 
ment at termination and those who did not are expected to bias 
the performance estimates because those who entered employment at 
termination are more likely to be employed at follow-up. It is 
assumed that those who were employed at termination are easier to 
locate than those who were unemployed because the interviewer has 
more contact sources (e.g./ name of employer). The resulting 
response bias can artificially inflate pjajrf ormance results at 
follow-up. 

To account for this problem, separate response rates should be 
calculated for those who were employed at termination and for 
those who were not. 

If the response rates of those employed at termination and those 
not employed differ by mere than 5 percentage points, then the 
"Worksheet for Adjusting Follow-up Performance Measures" in the 
Follow-up Technical Assistance Guide must be used to correct the 
follow-up measures. 
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Definitions of Terms Necessary 
for Completion of Reports 



EMPLOYMENT/TRAINING SERVICES 

ASSESSMENT — services are designed to initially determine 
each participant's employabi li ty , aptitudes, abilities and 
interests, through interviews, testing and counseling to 
achieve the applicant's employment related goals. 

FOLLOW-UP — IS the collection of information on a terminee's 
employment situation at a specified period after termination 
from the program. 

INTAKE — includes the screening of an applicant for eligi- 
bility and: (1) a determination of whether the program can 
benefit the individual; (2) an identification of the 
employment and training activities and services which would 
be appropriate for that individual; (3) a determination of 
the availability of an appropriate employment and training 
activity; (4) a decision on selection for participation and 
(5) the dissemination of information on the program, 

OUTREACH — activity involves the collection, publication and 
dissemination of information on program services directed 
toward economically disadvantaged and other individuals 
eligible to receive JTPA training and support services. 

EDUCATION STATUS 

LESS THAN HIGH SCHOOL — An adult or yoi;hh- (1) who is not 
attending school full-time and has not received a high school 
diploma or a GED certificate; or (2) who has not received a 
high school diploma or GED certificate and is enrolled 
full- tine in an elementary, secondary or postsecondary-level 
vocational, technical, or acadeaic school or is between 
school terms and intends to return to school. 

HIGH SCHOOL GRADUATE OR EQUIVALENT (No Post-High School) — 
An adult or youth who has received a high school diploma or 
GED Certificate, but who has not attended any postsecondary 
vocational, technical, or academic school. 

POST HIGH SCHOOL ATTEMOEE — An adult or youth who has 
received a high school ploma or GED certificate and has 
attended (or is attendiiig) any postsecondary-level 
vocational, technical, or academic school. 
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COLLEGE GRADUATE — A terminee who has received a degree 
(usually a BA or BS ) conferred by a four-year college, 
university or professional school or an advanced degree from 
one of these institutions. 

FAMILY_STATUS 

SINGLE HEAD OF HOUSEHOLD — A single, abandoned, separated, 
divorced or widowed individual who has responsibility for one 
or more dependent children under age 18. 

RACE/ETHNIC GROUP 

WHITE (NOT HISPANIC) — A person having origins in any of the 
original peoples of Europe, North Africa, or the Middle East. 

BLACK (NOT HISPANIC) — A person having origins in any of the 
black racial groups of Africa. 

HISPANIC — A person of Mexican, Puerto Rican, Cuban, Central 
or South American, or other Spanish culture or origin 
(including Spain), regardless of race. NOTE: Among persons 
from Central and South American coun triei" nly those who are 
of Spanish origin, descent, or culture should be included in 
the Hispanic category. Persons from Brazil, Guiana, and 
Trinidad, for example, would be classified according to their 
race, and would not necessarily be included in the Hispanic 
category. Also, the Portugese should be excluded from the 
Hispanic category and should be classified according to their 



?n f^^?^'''' OR ALASKAN NATIVE - A person having origins 

in any of the original peoples of North America, and who 
maintains cultural identification through tribal affiliation 
or community recognition. 

ASIAN OR PACIFIC ISLANDER - A person having origins in any 
tL^ original peoples of the Far East, Southeast Asia, the 
Indian subcontinent (e.g., India, Pakistan, Bangladesh, Sri 
Lanka, Nepal, Sikkim, and Bhutan), or the Pacific Islands. 
This area includes, for example, China, Japan, Korea, the • 
Philippine Islands, and Samoa. Hawaiian natives are to be 
recorded as Asian or Pacific Islanders. 

OTHER BARRIERS TO EMPLOYMENT 

LIMITED ENGLISH LANGUAGE PROFICIENCY — Inability of an 
applicant, whjse native language is not English, to 
communicate in English, resulting in a job handicap. 

HANDICAPPED INDIVIDUAL - Any individual who has a physical 
or mental disability which for such individual constitutes or 
results m a substantial handicap to employment. NOTE: This 
definition includes disabled veterans for reporting purposes. 
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READING SKILLS BELOW 7th GRADE LEVEL -- An adult or youth 
assessed as having English (except in Puerto Rico) reading 
skills below the 7th grade level on a generally accepted 
standardized test. 

NOTE: The following other methods of determination may be 
used : 

° A school record of reading level determined within the 
last 12 months. 

** If an applicant is unable to read and therefore cannot 
complete a self-application for the JTPA/EDWAA program, 
s/he may be considered to have English reading skills 
below the 7th-grade level. 

Individuals with any of the following may be considered 
to have English reading skills above the 7th-grade 
level : 

- A GED certificate received within the last year. 

- A degree (usually a BA or BS) conferred by a 4-year 
college, university or professional school. 

If there is any question regarding reading ability, a 
standardized test should be administered. 

BENEFITS STATUS 

UNEMPLOYMENT COMPENSATION CLAIMANT — Any individual who has 
filed a claim and has been determined monetarily eligible for 
benefit payments under one or more State or Federal 
unemployment compensation programs, and who has not exhausted 
benefit rights or whose benefit year has not ended. 

LABOR FORCE STATUS 

UNEMPLOYED: 15 OR MORE WEEKS OF PRIOR 26 WEEKS — An 
individual who is unemployed at the time of eligibility 
determination and has been unemployed for any 15 or more of 
the 26 weeks immediately prior to such determination, has 
made specific efforts to find a job throughout the period of 
unemployment, and is not classified as "Not in Labor Force", 

VETERAN STATUS 

VETERAN — An individual who served in the active military, 
naval, or air service (of the U.S.), and who was discharged 
or released therefore under conditions other than 
dishonorable. 

NOTE: The term "active" means full-time duty in the Armed 
Forces, other than duty for training in the reserves or 
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National Guard. Any period of duty for training in the 
reserves or National Guard, including authorized travel, 
during which an individual was disabled from a disease or 
injury incurred or aggravated in the line of duty, is 
considered "active" duty. 

VIETNAM-ERA VETERAN — A veteran, any part of whose active 
military, naval, or air service occurred between August 5, 
1964 and May 7, 1975. 

PROGRAM COSTS 

ACCRUED EXPENDITURES ~ The allowable charges incurred during 
the program year to date requiring provision of funds for: 
(1) goods and other tangible property received; and (2) costs 
of services performed by employees, contractors, 
subrecipients , and other payees. 

NOTE: These charges do not include "resources on order", 
i.e., amounts for contracts, purchase orders and other 
obligations for which goods and/or services have not been 
received . 

RETRAINING ACTIVITY 

BASIC EDUCATION — Includes remedial reading, writing, 
mathematics and/or English for non-English speakers. 

OCCUPATIONAL SKILLS TRAINING — Includes vocational education 
which is designed to provide individuals with the technical 
skills and information required to perform a specific job or 
group of jobs. For reporting purposes excludes On-the-Job 
Training . 

ON-THE-JOB TRAINING — Is training in the public or private 
sector given to an individual, who has been hired first by 
the employer, while s/he is engaged in productive work which 
provides knowledge or skills essential to the full and 
adequate performance of the ]ob. 
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Example of State Follow-up Questionnaire 

New Hampshire Follow-up Questionnaire* 





*Ncw Hampshire follow-up is conducted by Employment Training Institute (ETI). 
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HELLO* MAY I SPEAK WITH 



7 (Naae on cover sheet) 



HELLO. MY NAME IS (first ind Itst). I«M TAKING A SURVEY FOR THE NEW 

HAMPSHIRE JOB TRAINING COUNCIL AND - 

...I WOULD LIKE TO ASK YOU SOME QUESTIONS ABOUT THE JTPA PROGRAM AND SEE WHAT YOU 
HAVE BEEN DOING SINCE YOU LEFT THE PROGRAM. 

(For those «ho received « letter) 

...WITHIN THE LAST WEEK OR SO. YOU SHOULD HAVE RECIEVED A LETTER EXPLAINING 
THE PURPOSE OF THIS SURVEY. 

(For DES clients) 

...ro LIKE TO ASK YOU A FEW QUESTIONS ABOUT WHAT YOU HAVE BEEN DOING SINCE 
YOU WORKED WITH THE EMPLOYMENT SERVICE IN FINDING A JOB. 

WHAT YOU SAY IS VERY IMPORTANT TO US. YOUR ANSWERS WILL BE SUMMARIZED ALONG 
WITH HUNDREDS OF OTHERS TO HELP IMPROVE NEW HAMPSHIRE'S JOB TRAINING PROGRAMS. 
CALL OF YOUR ANSWERS WILL BE KEPT CONFIDENTIAL.] 



(If respondent doesn't recognize or adalt that he/she «as enrolled In a Job 
Training Prograa. continue with Question 1.) 



(If respondent Indicates that this Is an Inconvenient tim%$ sake an appointment to 
call hWher back.) 



Q 1 R 1 Appendix C - 1 

ERIC 



SECTION 1 - IX)L REQUIRED! 

A. I WANT TO ASK YOU ABOUT TWE WEEK STARHNG ON SUNDAY, _/ / , AND ENDING ON 
SATURDAY, / /_ WHICH WAS (LAST WEEK - 2 - 3 - 4 WEEKS AGO). 



© 



DID YOU DO ANY WORK FOR PAY DURING THAT WEEK? 
Yes 

No (Go to P*rt C) 




101 



(u^ HOW MANY HOURS DID YOU WORK IHAT WEEK? 
Hours 

(T) HOW MUCH DID YOU GET PAID PER HOUR IN THAT WEEK? 

S Dollars per hour 

102 $ Total Veekly Earnings. 

(Zh) HOW MUCH EXTRA, IF ANY, DID YOU EARN IN THAT WEEK FROM TIPS, OVERTIME, 
^ BONUSES, COMMISSIONS OR ANY WORK YOU DID ON THE SIDE, BEFORE DEDUCTIONS? 

102A $ Dollars 

^' fi^L^.i^'^ ^^^^^^ ^ ^^^^^^ SUNDAY, _/_, TO 

SATURDAY # / » 

©JM^-rSi'l "^^ ^^^^ ^"^"^ '^^^^^ "0** MANY WEEKS DID YOU WORK AT 

ALL FOR PAY DURING THE 13 WEEK PERIOD? 

103 Weeks 

^' Jm..^.°v*° question 1: NOW I WANT TO ASK YOU ABOUT THE EKTIRE 13 WEEKS FROM 
SUNDAY, _/_, TO SATURDAY, / . 

DID YOU DO ANY WORK FOR PAY DURING THE 13 WEEK ..PERIOD? 

Yes (Go to Q. 5) 

No 

104 

©HOW MAY WEEKS DID YOU DO ANY WORK AT ALL FOR PAY DURING THAT 13 WEEK 
PERIOD? 

105 Weeks 

MAY I HAVE YOUR DATE OF BIRTH FOR OUR RECORDS 
106 / / 
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SECTION 2 - OPINIONS ABOUT THE JOB TRAINING PROGRAM 

2?lcuT«r4'^^ ^° QUESTIONS ABOUT YOUR EXPERIENCE WITH THE NEW 

HAMPSHIRE JOB TRAINING PROGRAM. 



o 



HOW DID YOU FIND OUT ABOUT THE PROGRAM? 

1 - Friend/ relative told me 

2 - Saw advertisement In paper 

3 - Welfare worker 

4 - School counselor 

5 - Vocational Rehabilitation Counselor 

6 - Employment Service Counselor 

7 - Other 



207 

WHAT WAS THE MAIN REASON WHY YOU ENROLLED IN THE PROGRAM? 



208 

© 



1 - To get a job 

2 - To learn a skill or trade 

3 - To get a better Job 

4 - To get a GEO (High School Equivalency Diploma) 

5 - To got off Welfare 

6 - Other 



209 



V ^To^t^Jr^ol^^^^^^ ^^'^^ "^^^^ EXPERIENCE IN THE NEW HAMPSHIRE JOB 

TRAINING PROGRAM? WOULD YOU SAY IT WAS ... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



NOW I'D LIKE YOU TO GIVE ME YOUR OPINION ABOUT SEVERAL ASPECTS OF THE PROGRAM. 

® YOU FEEL V^IS^ ^'^^^^^ APPLYING FOR AND GETTING IN THE PROGRAM. DO 
210 ^ ^ " ^"^^^ 3 - Only Fair 4 - Poor 5 - No Opinion 
(If Poor) WHY DO YOU FEEL THAT WAY? 



211 



© 



IV rl^llrJIl^ ^'^^^"^ EVALUATING YOUR SKILLS AND PREPARING A SPECIFIC 
^ TRAINING PLAN FOR YOU. DO YOU FEEL IT WAS... 

^ ■ 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 
Vl3/ (If Poor) WHY DO YOU FEEL THAT WAY? 

213 



1R3 
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©HOW ABOUT THE QUALITY OF TRAINING YOU RECEIVED (FROM THE PROGRAM/FROM 
YOUR EMPLOYER? WAS IT... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 

214 



\J WHY 



Poor) 

DO YOU FEEL THAT WAY? 



215 



®(If OJT and If worfcod during referenced week) 
ARE YOU STILL WITH THE SAME EMPLOYER? 



Yes (Go to Q. 19) 

No 



216 



U7j HOW WAS THE HELP IN FINDING A JOB AFTER YOU COMPLETED THE PROGRAM? WAS 



1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



217 



^ydf Poor) WHY DO YOU FEEL THAT WAY? 
218 



©HOW WAS THE HELP YOU RECEIVED FROM THE JOB TRAINING STAFF WHILE YOU WERE 
IN THE PROGRAM? WAS IT... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



219 



^^(If Poor) WHY DO YOU FEEL THAT WAY? 
220 



©OVERALL, WOULD YOU SAY THAT THE HELP YOU RECEIVED FROM THE JOB TRAINING 
PROGRAM HAS IMPROVED YOU ABILITY TO FIND AND KEEP A JOB? 



1 - Yes 2 - No 3 - Don't Know 



221 



(22^ WHAT WAS THE BEST PART OF THE JOB TRAINING PROGRAM IN YOUR OPINION? 



222 



If: 4 
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(5^ WHAT'S : r. )NE THING ™T YOU WOULD LIKE TO SEE CHANGED IN THE PROGRAM? 
Codes for What you would like to see changed in the program: 

1 - "Don't Know" 

2 - "Nothing" (If followed by substantive response* code it) 

3 - Better instruction/teachers 

4 - Longer courses/vrograms 

5 - Provide more financial support during training (e-g., more pay, 

better pay) 

6 - Other 

______ 

SECTION 3 - WORK HISTORY 

NOW I'D LIKE TO ASK YOU SOME QUESTIONS ABOUT ANY JOBS YOU HAVE HELD SINCE LEAVING 
THE PROGRAM, 



© 



24j (If enplcyed during reference week:) 

ARE YOU STILL EMPLOYED? 

(If une^1«ved during reference weeks) 

ARE YOU CURRENTLY WORKING? 

Yes 



No (Go to Puge 7, (L 30) 



324 



© 



25 J WHAT IS THE NAME OF THE COMPANY TWAT YOU ARE WORKING FOR? 
(If More than one enployer* 11«t Pri—ry B^>iayer) 



»25 



© 



26 J WHAT KINO OF WORK DO YOU DO? 

(Probe) WHAT ARE YOU MOST IMPORTANT DUTIES: 



326A Title: 



326B DOT Code: 



© 



27 J 00 YOU WORK FULL TIME OR PART TIME? 

327 



(28^ HOW MANY HOURS PER WEEK DO YOU WORK? (If range, give Midpoint) 

hours 

328 



ins 
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29 
329 $ 



HOW MUCH ARE YOU MAKING PER HOUR? 



© 



330 



HOW MANY DIFFERENT JOBS HAVE YOU HELD SINCE LEAVING TWE PROGRAM? 
(If Zero* go to Page S$ (L 55) 



[Complete the following Job Profile for up to 3 Jobs* starting with most recent 
(not current) Job] 

MOST RECENT 

331 JOB TITLE 

332 DOT CODE 



333 START DATE 

334 END DATE 

335 DURATION 



Days - Weeks - Months 



336 HOURLY WAGE $. 



337 1 - FULL TIME / 2 - PART TIME 

338 REASON FOR LEAVING 



NEXT MOST RE(»{r 

339 JOB TITLE _ 

340 DOT CODE _ 



341 START DATE 

342 END DATE 

343 DURATION 

344 HOURLY WAGE $. 



J, 

J. 



Days - Weeks - Months 



345 1 - FULL TIME / 2 - PART TIME 

346 REASON FOR LEAVING 



NEXT MOST RECENT 

347 JOB TITLE _ 

348 DOT CODE _ 

349 START DATE _ 

350 END DATE . 

351 DURATIOK 



352 HOURLY WAGE S 

353 1 - FULL TIME / 2 - PART TIME 

354 REASON FOR LEAVING 



Days - Weeks - Months 



REASON FOR LEAVING CODES: 

1 - Laid off 

2 - Fired 

3 - Did not like work 

4 - Not enough pay 

5 - Too hard to get there (transp.) 

6 - Company went out of business 



7 - Did not like supervisor 

8 - Company moved 

9 - Decided to return to 

school /training 

10 - Found a better job 

11 - Health reasons 
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SECTION 4 - MISCELLAfCOUS 



(Ask Quastlons 55 to 59 only If currofitly uMaployod.) 



® 



ARE YOU CURRENTLY LOOKING FOR A JOB? 

Yes (Go to (L 57) 

No 



455 



© 



WHAT IS THE MAIN REASON THAT YOU ARE NOT LOOKING FOR A JOB? 

1 - Pregnant/ responsible for child care 

2 - Returning to school /training 

3 - Health 

4 - On temporary Lay-off 

5 - Moving from area 

6 - Waiting for new job to start 

7 - Have no transportation 

8 - Can't find a Job/have given up 

9 - Other 



456 (Go to (L 60) 



© 



WHAT TYPE OF JOB ARE YOU LOOKING FOR? 



457A Title: _ 
457B DOT (kxle 



0 



HOW ARE YOU GOING ABOUT LOOKING FOR A JOB? 

1 - Looking In the newspapers 

2 - Employment Service 

3 - Private employment agency 

4 - Asking friends 

5 - New Hampshire Job Training Council 

6 - Other ^ 



458 




IN LOOKING FOR A JOB, HAVE YOU BEEN ABLE TO USE THE JOB SEARCH 
TECHNIQUES OFFERED BY THE NEW HAMPSHIRE JOB TRAINING COUNCIL OR NOT? 

1 - Yes 2 - No 3 - Don't Know 



459 
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®tllllLJ;?^/TII'^?.S,^.,^?°^'- TRAINING PROGRAM SINCE LEAVING THE NEW 
HAMPSHIRE JOB TRAINING PROGRAM? 

Yes 



460 



461 



© 



® 



462 



© 



463 



464 



® 



465 



No (Go to Q- 62) 



WHAT TYPE OF SCHOOL OR TRAINING PROGRAM WAS IT? 



^iiSi S°rkl RECEIVE FINANCIAL SUPPORT FROM GOVERNMENT PROGRAMS, SUCH AS 
FOOD STAMPS, MEDICARE, UNEMPLOYMENT INSURANCE, OR WELFARE. ARE YOU 
RECEIVING ANY TYPE OF GOVERNMENT FINANCIAL SUPPORT? 

Yes 



No (Go to (L 64) 



WHAT KINDS OF ASSISTANCE ARE YOU RECEIVING? 
(Code all responses) 

1 - Food Stamps 

2 - Welfare 

3 - Unemployment Insurance 

4 - Medicaid 

5 - Day Care Assistance 

6 - Housing Subsidies 

7 - Other 



WOULD YOU RECOMMEND TWE JOB TRAINING PROGRAM TO A FRIEND OR RELATIVE? 
^ ■ Yes 2 - No 3 - Don't Know 



JHANK YOU FOR YOUR HELP ON THIS SURVEY. WE'LL BE CALLING YOU AGAIN IN 3 

1 - Yes "THANK YOU AND GOODBYE." 

2 - No (Go to (L 66) 

3 - Don't Know (Go to (L 66) 



©WOULD YOU PLEASE GIVE US A PHONE NUMBER WHERE WE WILL DEFINITELY BE 
JBLE JO^REACH YOU A FEW MONTHS FROM NOW? FOR EXAMPLE, THE PHONE NUMBER 
OF A FRIEND OR RELATIVE WHO WOULD KNOW WHERE WE COULD REACH YOU? 



466 Phone No. or 
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Sample RFP for FoIIow-Up Contractor 





*This sample RFP was adapted from RFPs used by the State of South Carolina, Budget and Control Board, 
Division of General Services; and by the Rhode Island Job Training Partnership Act Office, 
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Job Training Partnership Act Office 



Request for Proposal 

Proposal Requested 

The [State/SDA agency] seeks proposals froin qualified contractors for conducting 
a 13-week follow-up of JTPA participants. Work will include weekly telephone 
(or in person) interviews, mail questionnaire follow-up as required, data entry and 
verification, data analysis, and quarterly report preparation. The proposal must 
demonstrate that the follow-up activities meet the requirements of the U.S. 
Department of Labor Job Training Partnership Act data collection and reporting 
requirements, as specified in the Technical Assistance Guide for FoUow-Up and 
Validation (Attachment 1). Proposals will be evaluated on the basis of adherence 
to DOL requirements, bid specifications, technical approach, related experience, 
and cost. 

Response Deadline 

Proposals are to be submitted to the [State/SDA agency] no later than [time, 
date]. [Number of copies] of the proposal must be addressed to the attention of 
[name, address]. Interested bidders are invited to attend a preproposal 
conference on [time, date] at [agency, address]. Call [phone number] to confirm 
attendance or for additional information. 

Eligible Bidders 

Any governmental, non-profit, or private for-profit organization may apply. 
However, organizations that administer or operate JTTA programs serving Title 
II-A, or EDWAA participants in [State, SDA] are not eligible. 

Narrative Instructions 

The proposal narrative should explain how the bidder will provide the services 
requested in this RFP. In addition, it should demonstrate the bidder's applicable 
experience and ability to complete the survey research work required. All 
information presented in the proposal should respond clearly and concisely to the 
proposal specifications and should demonstrate that the bidder understands and is 
capable of fulfilling the responsibilities outlined in Section 6. 

Period of Performance 

The contract will be for a twelve (12) month period beginning [start date] and 
ending [end date]. The [State/SDA agency] may extend the contract if it appears 
to be in the best interest of the State/SDA and is agreeable with the Contractor. 
Such an extension may be less than, but will not exceed [maximum number]. The 
maximum contract per year is [$ amount]. 
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6. Contractor Responsibilities 



a. Select and Train Interviewers and Supervising Staff 

The contractor has sole responsibihty for selecting staff and for training 
and supervising staff in locating sample members, conducting interviews, 
recording contact attempts and results. The contractor must provide the 
[State/SDA agency] with the names and qualifications of staff assigned to 
the project, and their position titles and social security numbers. 

b. Schedule Interviews 

The contractor must develop and follow a schedule for conducting 
interviews that meets with the DOL guidelines. Interviews must be 
conducted between the 14th and 17th weeks following termination from 
JTPA. 

c. Conduct Interviews 

• Multiple attempts must be made and documented to contact each 
individual in the sample. Attempts will include telephoning the 
terminee at various times of day, telephoning the backup contacts, using 
directory assistance resources, and where necessary contacting other 
resources such as the JTPA service provider or last-known employer. 

• The contractor must make arrangements to receive return telephone 
calls on a toll-free (800 number) or collect basis from terminees in the 
sample. 

• The contractor must send letters and questionnaires to terminees in the 
sample who cannot be located by the end of the [-th week] after 
termination. 

• Interviews must be conducted directly with the terminee. Noone else 
(e.g., relatives, friends) may provide answers to the follow-up questions, 
although they may provide assistance in locating the terminee. 

d. Record and Verify Data 

The contractor will record contact attempts and responses to follow-up 
questions on the interview form following the coding scheme provided. 
The contractor will double enter the data into the MIS to ensure accuracy 
and validity. 
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Achieve Minimum Response Rates 

The contractor will achieve a 70% or better response rate for each of the 
following terminee groups: 

• Title II-A Adults in each SDA - employed at termination 

• Title n-A Adults in each SDA - not employed at termination 

• Title II-A Adult Welfare Recipients in each SDA - employed at 
termination 

• Title II-A Adult Welfare Recipients in each SDA - not employed at 
termination 

• EDWAA Participants in each SSA - employed at termination 

• EDWAA Participants in each SSA - not employed at termination 

Validate Responses to Ensure Accurate Data 

The contractor will verify that each respondent has been contacted by 
providing the respondent's date of birth. This date will be checked with 
[State/SDA] MIS records. 

In addition, the contractor will propose specific methods to ensure the 
accuracy and validity of the follow-up information collected, including 
monitoring, supervisory, and other appropriate approaches. 

Report FoIIow-Up Data 

• The contractor will report response rates for each of the groups listed in 
above on a [specify period] basis. Each [specify period] the contractor 
will transmit the follow-up data in the form of [MIS format] and submit 
the completed interview forms and any incomplete forms for individuals 
who have not mailed back responses by the end of the 18th week after 
termination. 

• At the end of each quarter, the contractor will provide summary follow- 
up reports for the [State/each SDA] for each participant category (Title 
II-A Adults, Title II-A Welfare Adults, EDWAA). Reports should 
include at a minimum the following information: 

(1) Terminees in the sample 

(2) Terminees contacted (by telephone/by mail) 
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(3) Response rates for each group ■ employed at termination and not 
employed at termination 

(4) % terminees employed at follow-up • employed at termination and 
not employed at termination 

(5) For Title II-A/JASR, average weekly earnings during 13th week (of 
employed terminees) - employed at termination and not employed 
at termination (For EDWAA, average hourly wage) 

(6) Average number of weeks worked during 13-week period - 
employed at termination and not employed at termination 

(7) [Add any other follow-up information collected by State/SDA] 

These reports will be sent to the [State/SDA-if State, also sent to each 
SDA]. 

7. [State/SDA] Responsibilities 



a. Select FoUow-Up Sample 

The [State/SDA agency] will draw random, fixed-rate samples of JTPA 
participants whose enrollment terminated each calendar week, beginning 
with [start date] and ending with [end date]. Samples will be drawn for 
Title II-A Adults, Title H-A Welfare Adults, and EDWAA terminees. 

The total weekly sample size is expected to average [#], but may vary from 
[range]. Approximately [#] of terminees will be followed-up each year. 

b. Provide Follow-Up Questionnaire 

Attachment 2 is the follow-up interview form, suggested introduction, and 
acceptable probes. 

c. Provide List of Follow-Up Sample and Contact Information 

The [State/SDA agency] will provide a computer printout listing each 
terminee in the sample, and at least two backup contacts. This information 
will be provided at least two (2) weeks before follow-up contacts with that 
terminee are to start. 

d. Provide Additional Assistance in Locating Terminees 

The [State/SDA agency] will make arrangements for JTPA service 
providers and OJT employers to assist the contractor in locating terminees 
in the sample who cannot be reached during the first two (2) weeks of the 
follow-up period. 
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e. Provide Technical Assistance 

The [State/SDA agency] will provide technical assistance to the contractor 
including the DOL follow-up requirements, training program descriptions, 
and a copy of the Technical Assistance Guide for FoUow-Up and 
Validation (Attachment 1). 

Proposal Content 

The proposal should include the following components: 

a. Procedures for completing tasks outlined in Section 6 above 

b. Capabilities of the contractor 

c. Budget with costs per contact, justified in terms of project activities and 
major expenditures. 

Selection Criteria 

Each proposal will be evaluated on the degree to which the contractor has 
addressed the specifications in this RFP, the related experience of the contractor, 
and the proposed budget. 
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How to Select a Random Sample 



The purpose of random sampling for the JTPA follow-up is to ensure that every 
terminee in each group sampled (II-A Adults, II-A Adult Welfare, III, and EDWAA) has 
an equal chance of being selected for the sample. The process of random sampling is 
straightforward, but requires care and accuracy to ensure that no bias is introduced. 

The most conventional and efficient tool for selecting a random sample is a 
random number table. A sample random number table appears in Exhibit C.l. Similar 
tables are available in most introductory statistics books and in collections of scientific 
tables. Also, many computer systems provide a routine that calculates random numbers. 

The special property of random numbers is that they are independent of «ach 
other. That is, each digit or number has an equal chance of being selected next, no 
matter what number was selected before. For example, if the first number selected is 
five, there would be an equal chance of getting any number next, incluoing a five. Using 
a random number table to select a sample ensures that the selection of one terminee is 
just as likely to occur as the selection of another. 

Several procedures may be used to select a random sample. Five procedures are 
described below. The first is automatically performed by the computer; the other four 
are manual. 

Procedure 1 • Automated MIS Selection 

Many MIS systems are capable of selecting random samples automatically. If 
these capabilities conform to generally accepted statistical practices for drawing samples, 
you can use them easily to select random samples for each group of terminees eligible 
for follow-up. 



17G 
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Exhibit E.l 



Random Number Table 



One thousand random digits 



Colum number 



dne 
mber 


I- 


10 


11- 


20 


21 


'30 


31- 


40 


41 


-50 


1 


48453 


83321 


51097 


62267 


11801 


98016 


16292 


25447 


13088 


58586 


2 


28888 


93145 


99052 




42475 


•J w / 7 X 


34068 




1585Q 




3 


00672 


54389 


94930 


48359 


88791 


28612 


91819 

7 XO X 7 


28478 


38^21 


878^8 

O / OHO 


4 


16377 


40682 


06927 


19063 


78214 


68719 


36349 


7'*n57 
/ jyjj 1 




f,^ 1 78 
oxx / o 


5 


62923 


44829 


53414 


40567 


90322 


6QQ65 


74881 


78501 




jyjjo X 


6 


87791 


92166 


04036 


79327 


36771 


38182 


19312 


57882 


46373 


58787 


7 


84140 


76746 


92624 


16333 


69654 


31164 


09929 


00856 


53382 


92281 


8 


93051 


92816 


19703 


13278 


21939 


41750 


46815 


34755 


84802 


74573 


9 


86438 


91275 


89794 


11268 


82425 


28264 


59675 


55166 


38589 


22255 


10 


65585 


97847 


50996 


44998 


20824 


83918 


23122 


86252 


58399 


87874 


11 


79782 


96739 


40676 


51209 


01483 


69407 


37866 


85225 


15070 


84730 


12 


77028 


74590 


39227 


81418 


04191 


02715 


90163 


45956 


77515 


13251 


13 


96729 


58766 


88856 


41533 


60605 


52564 


53386 


48399 


80880 


95670 


14 


29688 


40534 


94017 


28728 


16757 


58906 


08922 


51430 


09461 


73608 


15 


06894 


81782 


73549 


30787 


00584 


13065 


91879 


69108 


58548 


20338 


16 


22155 


01471 


38289 


30201 


69312 


16013 


53975 


62224 


80687 


85778 


17 


38151 


02407 


78419 


85166 


42116 


03059 


15525 


68969 


60304 


07692 


18 


47031 


96935 


15467 


55925 


13556 


51354 


13623 


31167 


82089 


27590 


19 


62635 


35059 


95261 


39375 


75985 


51099 


78495 


76524 


49133 


29303 


20 


31630 


92345 


59647 


62634 


56886 


45408 


79105 


65756 


39667 


22319 
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Procedure 2 




Step 1: Determine the proportion of terminees to be included in the sample. For 
example if the planned number of terminees is 1,700, the minimum sample 
size is 338. The proportion of terminees included in the sample is 
338/1,700 - .199. If a decision has been made to oversample (i.e., draw 
more than 338 terminees) the proportion would be higher. 

Step 2: Determine the number of terminees to be included in the sample for the 
period for which the sample is being drawn. For example, if in a month 
there are 104 terminees, and the sampling proportion is .199, the number 
in a monthly sample would be 104 X .199 = 21 (when rounded). 

Step 3: Assign a 3- or 4- digit random number to each terminee eligible for 
follow-up. Note that the random number table is one long string of 
numbers. To select a series of 4-digit random numbers, randomly pick a 
starting point in the middle of the table. From that point, simply go 
through the list, four digits at a time. 

Step 4: Reorder the list according to the assigned random 
numbers. 

Step 5: Select as the sample the first x individuals on the list, where x is the 

desired sample size for the week or month. In the example above, the first 
21 terminees would be selected. 

Procedure 3 

Step 1: Determine the proportion of terminees in the sample as in Step 1 of 
Procedure 2. 

Step 2: Assign a 3-digit random number to each terminee eligible for follow-up for 
the week or month. 

Step 3: Choose for the sample terminees whose random number is less than the 
sampling proportion times 1,000. For example, if the sampling proportion 
is .199, choose terminees with numbers between 000 and 198~that is, 
numbers less than .199 X 1,000 = 199. 
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Procedure 4 



Step 1: Determine the proportion of terminees in the sample 
as in Step 1 of Procedure 2. 

Step 2: Use the last three digits of the terminees' social 
security numbers to identify them. 

Step 3: Choose a 3-digit random number, r. Select for the sample all terminees 
whose last three digits of their social security number are between r and r 
+ (1,000 times the sampling proportion). For example, if the random 
number is 372 and the sampling proportion is .199, take all terminees 
whose last three digits are between 372 and (372 + 199 =) 571. If the 
random number plus (1,000 times the sampling proportion) is over 999, 
take terminees whose last three digits are above the random number and 
terminees whose last three digits are below the random number plus (1,000 
times the sampling proportion) minus 1,000. For example, if the random 
number is 932, take all individuals from 932 to 999 and those from 000 to 
(932 + 199 - 1000 = ) 131. 

Procedure 5 

Step 1: Order the terminees eligible for follow-up according to the last several 
digits of their social security numbers. 

Step 2: Determine the ratio of the size of the group eligible for follow-up to the 
minimum sample size. Calculate the integer proportion for that ratio; call 
this integer n. For example, if total planned terminees is 1,700, the 
minimum sample size is 338 (from Exhibit 2.1), the ratio is 1,700/338 = 
5.02, and the integer portion is 5. 

Step 3: Randomly select a random number, r between 1 and n. 

Select for the sample the rth terminee in the ordered list and every nth 
terminee afterwards. 
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Selected Survey Research References 

This list includes two groups of references designed to guide States, SDAs, and 
contractors involved in JTPA follow-up: 

1. general survey research references ■ for guidance on overall design issues, 
methods of anaysis, and appropriate uses of the data collected. 

2. survey design and implementation references - for guidance on specific 
techniques for collecting follow-up data collection. 



General References 

Babbie, Earl R. (1973). Survey research methods. Belmont, CA: Wadsworth Publishing 
Co. 

Backstromm, Charles H. and Hursh-Cesar, Gerald (1982). Survey research . 2nd ed. 
New York: John Wiley and Sons. 

Crane, William D. and Brewer, M. B. (1985). Principles nf so cial research and 
evaluation. Boston: Allyn-Bacon. 

Isaac, Stephen and Micahel, William B. (1981). Handbook in research and evahiatinn 
2nd ed. San Diego; Edits Publishers. 



Survey Design and Implementation References 

Blankenship, A. B. (1978). PrQfes.sional telephone surveys New York; John Wiley & 
Sons. 

Department of Manpower Development, Policy and Evaluation Division (December, 
1981). Designing and implementing local follo w-up systems . Parts I and II. Boston; 
Author. 

Department of Manpower Development, Policy and Evaluation Division (June, 1980). 
Methods an d strategies to locate a n d contact survey respondents: Technical 
assistance manual. Boston: Author. 
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Dillman, Don A, (1978). Mail and telephone surveys: The total design method . 
New York: John Wiley & Sons. 

Fowler, Floyd J., Jr. and Mangione, Thomas W. (1990). Standardized Survey 
Interviewing: Minimizing Interviewer -Related Error . Newbuiy Park, CA: Sage 
Publications. 

Frey, James H. (1983). Survey research bv telephone . Beverly Hills: Sage Publicaionts. 

Lewis, Morgan V. (1972). "Finding the hard-to-locate: A review of best practices." In 
Michael E. Bonis (ed.), Evaluating t he impact of manpower programs . Uxington, 
MA: Lexington Books, D.C. Heath and Co. 

Sudman, Seymour and Bradburn, Norman M. (1982). Asking questions: A practiga ) 
guide to qu estionnaire design . San Francisco: Jossey-Bass, Inc. 

Warwick, Donald P. and Liniger, Charles A (1975). The sample survey: Thenrv and 
practice . New York: McGraw-Hill. 
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Exhibit G.l 



Worksheet to Calculate DifTerences in Response Rates 



1. Total Title II-A Adults 

a. Employed at Termination 

b. Not Employed at Termination 



I 

Size of 

Selected 

Sample 



II III 
Numbei of Response 
Completed Rate 
Interviews (coi ii + Oxioo 



c. Difference in Response Rates; 

if >5%, Go to Bias Adjustment Worksheet, and on to Part 2. 
if ^5%, Go to part 2. 



2. Title II-A Welfare Adults (Includes welfare terminees contained in Total II-A Adult 
group, and any supplementary welfare terminees) 

a. Employed at Termination I 1 I 1 I 



b. Not Employed at Termination 



c. Difference in Response Rates; 
if >5%, Go to Bias Adjustment Worksheet, and on to Part 3. 
if ^5%, Go to part 3. 



3. Dislocated Workers (EDWAA) 
a. Employed at Termination 



b. Not Employed at Termination 



c. Difference in Response Rates; 
if >5%, Go to Bias Adjustment Worksheet, 
if ^5%, Finalize caluclations. 
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Exhibit G.l (continued) 



INSTRUCTIONS: Worksheet to Calculate Differences in Response Rates 

1. In Column I (Size of Selected Sample) enter the number of terminees sampled 
for follow-up this Program Year for each group as indicated. This may be 
githcr the minimum required .-.ample determined, a larger number, or the 
entire group of eligible terminees, as determined by the State or SDA. 

2. In Column II (Number of Completed Interviews) enter the number of 
terminees with completed follow-up interviews for each sample group as 
indicated. 

3. Divide the number of completed interviews, or respondents (Column II) by the 
sample size (Column I); then, multiply the result by 100 to calculate the 
response rate. Enter the response rate in Column III (Response Rate). Each 
of these response rates should be at least 70 percent. 

4. Subtract the response rate for terminees who were not employed at 
termination (Line b) from the response rate for terminees employed at 
termination (Line a), and enter the difference on Line c for each sample as 
indicated. 

Total Title II-A Adults 

5. If the difference in response rates between II-A adults who were and were not 
employed at termination exceeds five (5) percentage points, use the 
adjustment worksheet provided (Exhibit G.2: Worksheet to Adjust FoUow-Up 
Performance Measures for Nonresponse Bias) to calculate adjusted follow-up 
performance measures for this group. 

6. If the difference in response rates is less than or equal to five (5) percentage 
points, then follow the simplified calculations contained in these instructions to 
compute the follow-up performance measures for this group. Adjusting for 
nonresponse bias is optional in this case. 

7. Proceed to Parts 2 and 3. 

II-A Adult Welfare and EDWAA/Title III Dislocated Workers: 

8. The steps for Title II-A Adult Welfare and Total Adults and EDWAA/Title 
III Dislocated Workers are the same as those listed above. 
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Exhibit G.2 presents instructions for making adjustments where the difference in 
response rates between those who were employed and those who were not employed at 
termination exceeds five (5) percentage points. Instructions involve five basic entries, 
both for terminees employed and for terminees not employed at termination, taken from 
the State's or SDA's aggregated follow-up data: 

1. The number of terp inees eligible for follow-up for the Program 
Year. (This figure > not the number entered in Column I on the 
Response Rate Worksheet in Exhibit G.l, which is the number in 
the follow-up sample. Rather it is the total number of terminees 
eligible for follow-up from which the sample was drawn.) 

2. The number of respondents (completed interviews) from the sample 
for the same period (This should correspond to the number entered 
in Column II on the Response Rate Worksheet.) 

3. The number of respondents who were employed during the 13th 
follow-up week 

4. The average gross weekly earnings for those respondents employed during the 
i3th follow-up week 

5. The average weeks worked during the 13-week follow-up period for 
all respondents 

Using these entries, you can make all of the remaining computations to adjust the follow- 
up performance measures for differences in response rates. Again, using this adjustment 
procedure is optional where the response rate differential is five percentage points or 
less. 
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Workshttt to Adjust Follow-up Pwfonnance MMsurot for Nonrosponso Bias 



Gram Radptont: 
N«nt 

AddMM 

CNy/ZIp 



ADULT A YOUTH PROGRAM 

^TWall-AAduNi 

TMtll-AAdullWtNart 

e%Adulli 

eKAduHWtNM* 



DI810CATED WORKER PROGRAMS 

SuteiiiiArMPrognn 

8M9 Prapfanw 
Swy NHofwi rm«v» 



FMowupt Conductad From 
through 



(Report Dale) 




STATUS AT TERMINATION: 



Employod 
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Exhibit G.2 (continued) 



INSTRUCTIONS: Worksheet to AcUust FoIIow-up Performance 

Measures for Non-Response Bias 

1. Check the performance group (Title II-A Adult or Adult Welfare, EDWAA) 
for which the adjustment is being made - in the upper left-hand corner. 

2. In Column A, enter the number of terminees who were employed at 
termination, not employed at termination, and the sum of the two items, for 
the group eligible for follow-up. i 

3. In Column B, enter the number of terminees, for both those employed and 
those not employed at termination, for whom completed interviews were 
obtained at follow-up. 

4. Enter the respective follow-up information for both sets of respondents in 
Columns C, E, and G, as follows: 

Column C: the number of respondents employed in 
the 13th follow-up week 

Column E: the average gross weekly earnings for those respondents 
employed in the 13th follow-up week 

Column G: the average number of weeks worked during the 13-week 
follow-up period, for all respondents 

5. To calculate the adjusted follow-up employment rate: 

- Multiply the number eligible for follow-up (Column A) by the number of 
respondents employed at follow-up (Column C) separately for those who 
entered employment and those who did not enter employment at 
termination. Then divide the two products by the number of completed 
interviews in each group (Column B). 

- Enter the result in Column D for those who were and were not employed 
at termination. 

- Add the estimated number of employed in Column D and enter the total 
in Box J. 

- Divide the estimated number of employed terminees (Box J) by the total 
number of terminees eligible for follow-up (Box I), and multiply by 100 to 
obtain the adjusted employment rate at follow-up. 

- Enter the adjusted employment rate in Box IC 
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Exhibit G.2 (continued) 



6. To calculate the adijusted average gross weekly earnings: 

- Multiply the average weekly earnings of employed respondents (Column E) 
by the estimated number of employed terminees (Column D), separately 
for those who entered and those who did not enter employment at 
termination. 

- Enter the result in Column F for both sets of terminees. 

- Sum the estimated earnings of employed terminees in Column F and enter 
the total in Box L. 

- Divide the estimated earnings of employed terminees (Be - L) by the total 
estimated number of employed terminees (Box J) to calculate the adjusted 
average gross weekly earnings of terminees employed at follow-up. 

- Enter the adjusted average weekly earnings of employed respondents in 
Box M. 

7. To calculate the adjusted average number of vi^ks worked: 

- Multiply the average number of weeks worked (Column G) by the number 
of terminees eligible for follow-up (Column A), separately for those who 
were and those who were not employed at termination. 

- Enter the result for both sets of terminees in Column H. 

- Add the estimated average number of weeks worked in the 13-week 
follow-up period and enter the result in Box N. 

- Divide the estimated total number of weeks worked in the follow-up period 
(Box N) by the total number of terminees eligible for follow-up (Box I) to 
obtain the adjusted average number of weeks worked. 

- Enter the adjusted average number of weeks worked in Box 0. 

8. Finally, take the adjusted performance measures contained in Boxes K, M and 
O~for the Employment Rate, Average Earnings, and Average Weeks 
worked~and enter these on the appropriate lines on the JASR/WAPR. 
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Software to Support JTPA Follow-up* 



Overview 

The main function of the Software to Support JTPA Follow-up is to provide: 

• a script and automatic data entry form to be used by telephone 
interviewers conducting JTPA follow-ups. I 

To support follow-up interviewing, the software also provides the capabilities for: 

• entering, or loading from a DOS file, participant information useful 
in the interview process 

• archiving the interview results to a cumulative DOS file of follow- 
up data 

• generating a JASR-Iike report, based on the archived follow-up 
interview data 

The follow-up software does not include all functions related to follow-up, such as 
maintaining a log of contact attempts; and it is written for a limited range of hardware. 
Nevertheless, the software is intended to support the interview process and facilitate the 
collection and reporting of accurate and vahd JTPA follow-up data. 

The follow-up software is contained in a high density 5-1/4" diskette, /unerican 
Institutes for Research (AIR) staff programmed the software, which is written as a 
run-time application in the Paradox2 database language. Paradox2 is published by 
Borland, Scott's Valley, California. The software can be run as a stand-alone system, on 
an IBM-compatible PC with a 80286 or 80386 processor, a hard disk, and a high density 
disk drive. (Although the software can also run on a 8088 processor, it operates quite 
slowly.) 

The follow-up software is available on request from the DOL/ETA. To obtain a 
copy, please contact Steve Aaronson and his staff at l-(202)-535-0687. 



*This software is provided by AIR, which prepared this TAG. Its use and application are not required by the 
DOL, and its availability does not constitute a DOL endorsement. 
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Loading the Software 



To load the follow-up software, follow these steps: 



1. Make a new directory called JTPACATI: mkdir JTPACATI 

2. Enter the new directory: cd JTPACATI 

3. Copy the follow-up program from the 

diskette provided onto your hard disk: copy a:*.* 

4. Type "foUowup": followup 

The follow-up software menu screen will appear. 

[Note: The software provided includes 25 sample cases, with fictitious names, social 
security numbers, demographic, and termination data. You can practice using the 
software, using these fictitious cases, then delete the cases when you are ready to enter 
your own JTPA database.] 

Tlie follow-up software, like the Paradox language in which it is written, provides 
a series of menu options. One option on the Main Menu is Directions, which 
supplements the information presented here in Appendix H. Reading these directions is 
recommended. Using the software will also familiarize you with a Paradox idiosyncracy: 
the ubiquitous use of the function key F2 to proceed to the next step in a process. When 
in doubt, Press F2. (Do not press Escape to leave the program or any menu option.) 

The main menu has the following options, which are described in turn below: 



Main Menu 




Intervlw Direction* Enter Report Archive Quit 
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Interview 

Select this menu option when you arc ready to conduct a follow-up interview. 
The first step is to select the terminee you wish to interview rext, and the system 
provides a menu for the method of selection. 

Respondent Selection Menu 



New Partial Terminee Interviewer Browse Quit 



Prior to conducting interviews, information about JTPA terminees (at least their names, 
termination statuses, and termination dates) will have been entered or loaded, using the 
Enter option on the Main Menu. All terminees will have been marked "New," indicating 
that they have not yet been interviewed for the follow-up. When they are interviewed, 
their status changes to "Complete", or to "Partial" if the interview is begun but not 
completed. 

New. Selecting New locates the first remaining "New" case and brings that 
terminee's record onto the screen. At this time, or any time during the interview, you 
can examine all of the available data on this terminee by pressing the "\" (backslash) 
key. The next step is to type the terminee's name and to call the terminee, using the 
telephone number on the screen. 

Partial. Selecting Partial locates the first remaining partially completed case for 
interview. Otherwise it acts exactly like the New option. 

Terminee. Selecting Terminee locates the first case with a terminee name 
matching the name you type in. Otherwise it acts exactly like the New option. (Be sure 
to type in the terminee's name exactly as it was entered in the database.) 

Interviewer. Selecting Interviewer locates the first non-completed case in which 
the interviewer's name matches the name you typed in. For this option, it is expected 
tliat interviewers will have been pre-assigned using the Enter option on the Main Menu. 
After locating the appropriate case, the Interviewer option acts exactly like the New 
option. (Be sure to type in the interviewer's name exactly as it was entered in the 
database.) 
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Browse. Selecting Browse shows the entire Paradox follow-up table, which can be 
viewed using the keyboard cursor keys. You can select a particular terminee to interview 
or merely examine the existing records. If you wish to interview a particular terminee, 
you should place the cursor on the row in the table for that terminee, and type ";" (a 
semi-colon). If you wish to return to the menu instead, type "!" (an exclamation point). 

Once in the interview, from whatever method of entry, you will see a series of 
screens. The software selects the proper sequence of screens, depending on the 
information given. You do not need to worry about the sequence. To exit the interview 
at any time, press F2. 

On each interview screen, three types of information appear: 

1. Background information appears in reverse capital letters. 

2. The words to be read aloud during the interview appear in usual 
upper and lower case letters. The question wordings correspond to 
the wordings in this Guide. 

3. Information to be typed on the keyboard before continuing is 
indicated with a cursor. 

Note: For follow-up questions that require a Yes or No response (#1 and #6), 
type Y or N beside the appropriate response. 

[For programmers only: The script that determines the sequence of screens is called 
"PATHWAY.SC. To change the sequence of screens, you can edit the script using a 
word processor. Before editing the sequence, be sure that the old version is safely 
copied. To implement the change you must issue the command "PD0XRUN2 
PATHWAY" after editing PATHWAY.SC. However before issuing that command, re- 
copy the file "DOLPROCS.LIB" from the original diskette.] 

Directions 

Selectmg the Direction option brings up a series of screens with explanatory 
information about the other options available in the software. Press Enter for the next 
page, PageUp for the previous page, or F2 to return to the previous menu. 
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Enter 

Selecting the Enter option creates a table of participant information that can be 
accessed and updated during the follow-up interviews. This option should be selected on 
a periodic basis (e.g., weekly) when new cases are sampled for follow-up. The 
information can either be typed in by keyboard, or loaded from an ASCII (DOS) file 
that might be exported from your agency's MIS. You have the option of entering any or 
all of the information requested, and you can specify the layout for records in the DOS 
file. The only terminee infonnation that is used to determine the sequence of follow-up 
questions is whether the termination status is "Placed" (coded "P') or not. Also, the 
termination date is used to determine the appropriate reference week(s) to refer to in 
the questions. Although other information is optional, omitting items, such as the 
participant's name, causes an awkward blank in the screen script for the interview. 

Note that accessing a case under the Enter menu causes that case to be called 
New for interview purposes. 

After selecting Enter^ you will see the following menu, whose options are 
described below: 

Enter Menu 



Save Current Restore Table Empty Load from DOS Type in Data Quit 



Save Current. Select this option to save the current follow-up table before 
proceeding. The table is saved as a dBase m file and must be given a file name with the 
SLflix ".dbf. Becausw new information entered will write over the current follow-up, select 
this option when you want to be able to retrieve the current version of the follow-up table in 
the future. 

Empty. Select this option to empty the current follow-up table before proceeding. 
Be sure to save existing data (using the Save Current option) before using Empty. If you 
merely want to add more information about 'the terminees (in the same sort order!), then it 
is not appropriate to empty the cunent follow-up table before proceeding. In all other 
cases, it is. (Use Empty when you are ready to remove the 25 fictitious cases provided with 
the software.) 
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The use of the dBase III file format for temporary storage of follow-up tables allows 
users with dBase III but not Paradox to make flexible use of the interview results. The 
system is oriented to handle follow-up interviews in batches (such as weekly batches). The 
Archive function (see page H-7), on the other hand, allows you to aggregate the results of 
multiple follow-ups in DOS files; and the Report function (see page H-6) makes use of the 
archived DOS files. 

Restore Table. Select this option to restore a saved follow-up table before 
proceeding. The table is assumed saved as a dBase III file with suffix ".dbf . 

Load from DOS. Select this option if you wish to load terminee information from an 
existing DOS file. If you choose this option, you will be asked to examine and alter the 
specifications of columns and lengths for various fields. If a particular field is not on the 
file, you should skip that specification. After making changes and pressing F2, you will be 
asked to enter the name of the DOS participant file that is the source of information. After 
entering the file name, the information will be loaded into the follow-up table. You can 
then add more information manually by selecting Enter and not emptying the table before 
proceeding to the T^pe in Data option. 

T^pe in Data. Select this option to enter or change terminee data. In keying in data, 
you can repeat the value of the same field from the preceding case by typing (carat). 

The following fields provide terminee information: 



Participant 
Name 

Telephone Number 
Address, City, State, Zip 
Social Security Number 
Termination Status, Date 
Termination Employer, Wage 
Gender, Race, Age, 
High school diploma? 
Welfare recipient? 



Other 

Other Contact Name 

Other Contact Phone 
Training Program Name 

Interviewing Agency Name 
Interviewer's Name 

Mail Notice Date 
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Report 



By selecting Report, you can acquire the summary follow-up information needed for 
the JASR report. An ASCII (DOS) file created by the Archive function may accumulate 
results of batches over a prolonged period, such as quarterly or annually. After you specify 
the ASCII file, the follow-up software computes the summary statistics (sample size, 
response rate, follow-up employLtient rate, follow-up earnings, and follow-up weeks worked) 
for all cases and for cases that were welfare recipients. To comply with DOL regulations, 
the software automatically calculates the follow-up statistics and adjusts for the response- 
rate differential between participants placed and those not placed at termination. 

[For programmers only: The computations can be reviewed by examining the script in the 
DOS file named "O.SC, the essential part of which is shown below.] 



Script for JASR Follow-up Computations 



afs-afsp+af sn wf s-wfsp+wfsn 

if (afs) >0 then arr-round((100*(arrp+arrn)/(afsp+afsn)) ,1) 

else arr--l 
endlf 

if (wfs) >0 then wrr-round((100*(wrrp+wrrn)/(wfsp+wfsn)) ,1) 

else wrr--l 
endif 

afer-round(100*(afsp*aep /arrp+afsn*aen /arrn)/ 

(afsp*arrp/arrp+afsn*arrn/arrn) , 1) 
wfer-round(100*(wfsp*wep /wrrp+wfsn*wen /wrrn)/ 

(wfsp*wrrp/wrrp+wfsn*wrrn/wrrn) , 1) 
af te-round( (af sp*atep/aep+af sn*aten/aen) / 

(afsp*aep/aep +afsn*aen/aen ),0) 
wf te-round( (wf sp*wtep/wep+wf sn*wten/wen) / 

(wfsp*wep/wep +wfsn*wen/wen ),0) 
afww-round( ( af sp*awwp/arrp+af sn*awwn/arrn) / 

(afsp*arrp/arrp+af sn*arrn/arrn) , 1) 
wfww-round( (wf sp*wwwp/wrrp+wf sn*wwwn/wrrn) / 

(wfsp*wrrp/wrrp+wfsn*wrrn/wrm) , 1) 

Notes: The prefix "a" refers to all adults, "w" to welfare recipients. 

The suffix "p" refers to placed terminees, "n" to those not placed, 
"fs" is follow-up sample; "rr" is number or percentage of respondents; 
"fer" is follow-up emplo3anent rate; "fte" is follow-up earnings; and 
"fww" is follow-up weeks worked. Division by zero yields zero. 
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Note that no distinction is made in this system between JTPA Title II-A and 
EDWAA. The system can be implemented on separate copies of the follow-up software for 
the different titles. 



Archive 

Selecting Archive on the Main Menu allows you to save the results of the follow-up 
by adding information to a specifiable DOS (ASCH) file. The follow-up software makes use 
of DOS ASCII files because the system is designed for an environment in which other 
software may be used for information management and reporting. After selecting Archive, 
type in the name of the DOS file to which information is to be added. Then select J (for 
JASR) or N (for Notes) to indicate where the information should be stored. Two kinds of 
information are stored on different files for use with other software: 

1. For JASR reporting purposes, the follow-up responses are stored, along 
with the terminee's SSN, the date of the interview, and the interviewer's 
name. The only pre-follow-up information stored on this file is the 
information needed for the JASR computations: the termination status ("P" 
or other) and whether the terminee had been a Welfare Recipient at 
eligibility determination ("Y" or "N"). 

2. For other uses, a file is written with the terminee's SSN and three fields, 
on three successive records: "Notes" (a 240-byte field containing any 
information keyed by the interviewer, such as finding an error in 
termination information); "Problems" (a 240-byte field containing the 
terminee's response (if asked) to a question about problems obtaining 
employment); and "Advice" (a 240-byte field containing the terminee's 
response to a request for advice to improve the training program in the 
future). 

In order for you to access the Archive option repeatedly for status updates, selecting the 
Archive function will not alter the condition of the current follow-up table. However, the 
information is added to the end of the DOS file named, so you must be careful not to add 
multiple copies of the information to the archive file. For example, you may delete or 
rename an existing archive DOS file before archiving another. 
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Quit 

Selecting the Quit option exits the follow-up software program and returns you to 
DOS. All files remain intact, unless the hard disk crashes. Exiting from the follow-up 
software in any other way (e.g., by turning of the power) may lose some data. 
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ABSTRACT 

The goal of the Job Training Partnership Act (JTPA) 
is long-term employability . To obtain feedback on the effectiveness 
of JTPA programs in promoting retention in the labor force, the U.S. 
Department of Labor (DDL) requires states and service delivery areas 
(SDAs) to collect information on the postprogram outcomes of JTPA 
participants. This technical assistance guide helps states collect 
data. This guide helps all those involved in the JTPA follow-up 
process to obtain accurate, valid data that meet the federal 
reporting requirements, provide an equitable basis for incentive 
awards, and yield useful information for program management decisions 
at all levels. In addition, it presents a detailed description of 
follow-up and validation strategies gathered in a recent study 
conducted for the DOL. This study surveyed follow-up staff in 50 
states and the District of Columbia and Puerto Rico, plus 151 SDAs 
were surveyed to determine how follow-ups are conducted, how accuracy 
of data is ensured, and what strategies are used for follow-up data 
collection. The guide is intended to help users adapt existing 
procedures or adopt new strategies to improve information obtained. 
Eight references, 8 appendices, and 21 exhibits are included in this 
report. The appendices include: (1) JTPA Annual Status Report and 
Instructions; (2) Worker Adjustment Annual Program Report and 
Instructions; (3) Example of state Follow-up Questionnaire; (4^ 
Sample Request for Proposal for Follow-up Contractor; (5) How 
Select a Random Sample; (6) Selected Survey Research Reference^, (7) 
Procedures to Adjust for Nonresponse Bias; and (8) Follow-up 
Softwak-e. (NLA) 
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Preface 



Passage of the Job Training Partnership Act (JTPA) in 1982 provided an 
innovative approach to employment and training, designed to address an important social 
and economic issue: Can American society bring unemployed or hard-to-employ 
individuals into or back into the productive workforce through a partnership of industry 
and government aimed at job training? Unlike previous employment and training 
programs, JTPA is based on specific performance standards designed to assess the 
effectiveness of ongoing programs and to reward programs that are effectively achieving 
the goals of JTPA. 

The bottom-line goal of JTPA is long-term employability-participation in the 
workforce that goes beyond short-term placement in a job. To obtain feedback on the 
effectiveness of JTPA programs in promoting retention in the labor force, the U.S. 
Department of Labor (DOL) required States or SDAs to collect information on the 
postprogram outcomes of JTPA participants, beginning October 1, 1986. Since that time. 
States or SDAs have been collecting data about the employment status and earnings of 
JTPA terminees during the 13-week period after leaving the program. 

To help States and SDAs collect follow-up data, the U.S. Department of Labor, 
Employment and Training Administration (ETA), issued the Follow-up Technical 
Assistance Guide for Postprogram Data Collection (TAG) in June 1986. In Program 
Year (PY) 1988, some States began to include in their incentive award calculations the 
SDA's performance on one or more of four postprogram performance measures. 
Consequently, valid and accurate follow-up data became a critical concern to ensure that 
incentives are distributed fairly among SDAs. 

This Technical Assistance Guide for Follow-up and Validation (TAG) is an 
update of the earlier TAG. It is designed to help all those involved in the JTPA follow- 
up process obtain accurate, valid follow-up data that meet the federal reporting 
requirements, that provide an equitable basis for incentive awards, and that yield useful 
information for program management decisions at all levels. 

In addition to providing current information on JTPA follow-up requirements, 
the TAG presents follow-up and validation strategies gathered in a recent study 
conducted for the DOL by The American Institutes for Research (AIR), Palo Alto, 
California. Follow-up staff in 50 States plus the District of Columbia and Puerto Rico, 
and 151 SDAs were surveyed to determine how follow-ups are being conducted, what 
efforts are taken to ensure the accuracy of the data collected, and what strategies are 
helpful in obtaining valid follow-up data. A detailed description of this study, including 



methodology, results, and recommendations, appears in AIR's Final Report: Study of 
JTPA Postprogram Follow-up Validation. 

The TAG is not intended to be a complete set of instructions for designing and 
implementing a follow-up system. Rather, it is intended to help States, SDAs, SSAs, and 
independent follow-up contractors adapt existing procedures or adopt new strategies to 
improve the follow-up data collection and to enhance the accuracy, validity, and 
ultimately the usefulness of the information obtained. 
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Section 1 - Overview of Follow-up and Validation 

This overview includes an introduction to JTPA follow-up and validation and a 
description of the purposes and organization of this Technical Assistance Guide for 
JTPA Follow-up and Validation (TAG). 

1.1 Introduction 

Since October 1, 1986, States have been required to collect and report to the U.S. 
Department of Labor (DOL) selected postprogram data at the Service Delivery Area 
(SDA) level for all adult and adult welfare terminees served under Title II-A, and at the 
Statewide level for terminees served under Title III. Beginning in Program Year (PY) 
1989, States are required to collect similar postprogram data on a Substate Area (SSA) 
basis for terminees of the Economic Dislocation and Worker Adjustment Assistance Act 
(EDWAA) program. 

The postprogram data collection requirements are based upon three performance 
measures that describe the labor market experiences of former JTPA participants over 
the prescribed 13-week follow-up period. These measures include the employment rate 
during the 13th full calendar week after program termination; the average gross weekly 
earnings of employed terminees during the 13th full calendar week after program 
termination; and, for all respondents, the average number of weeks employed during the 
13-week follow-up period. 

Beginning in PY 1988, these performance measures were used to award monetary 
incentives to SDAs that exceeded their standards. To assure that incentives are awarded 
equitably and that the adjustment models are adequate, it is essential that the follow-up 
data be accurate and valid and that the procedures used to collect these data be 
consistent and thorough. Without assurance of accuracy, validity, and uniform data 
collection procedures across SDAs/SSAs* in each State, incentive funds may not be 
awarded fairly, and follow-up data may not confidently be used in program management 
and evaluation. 

Implementing an effective follow-up system that meets these criteria requires the 
commitment and motivation of every individual involved in the follow-up process-from 
the JTPA terminees who provide the information, to the local SDA/SSA or contractor 



*In subsequent sections of this TAG, SDA is gcneraUy used to refer to SSAs, as weU as SDAs. 
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staff who help to locate the participants, to the follow-up interviewers who collect the 
information, to the JTPA officials at all levels who report and use the information. 
These individuals must clearly understand the purposes of follow-up, appreciate the 
potential usefulness of the data, and be motivated to do what it takes to obtain complete, 
accurate, valid information from as many terminees as possible. Ongoing communication 
among these individuals is essential also to ensure the timely transmission of information, 
to produce maximum response rates, and to provide adequate monitoring and quality 
control that will assure accurate, valid 'results. 

Purposes of the TAG 

This TAG is a supplement to the Instructions for the JTPA Annual Status Report 
(JASR) and the Worker Adjustment Program Annual Status Report (WAPR). An 
update of the original Follow-up TAG for Postprogram Data Collection (Version 3.0, 
June 1986), it is designed to help States, SDAs, SSAs, and independent follow-up 
contractors to: 

• understand and apply the DOL follow-up requirements for data 
collection and reporting 

• implement effective strategies for obtaining high response rates and 
valid follow-up data 

• obtain accurate, valid follow-up data that provide the basis for the 
fair and equitable award of incentives 

• use the information effectively for Federal, State, and local program 
planning and evaluation 

Because States and SDAs have been developing and implementing JTPA follow- 
up since PY 1986, they have had an opportunity to learn what strategies have or have 
not been effective in obtaining valid follow-up data within their particular setting. This 
TAG will provide States and SDAs an opportunity to assess what has and has not 
worked in conducting follow-up, as well as an opportunity to consider adopting strategies 
that have been used effectively elsewhere as part of an ongoing effort to evaluate and 
improve JTPA program components. 
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Organization of the TAG 

The TAG is organized into the following four sections: 
Section 1 - Overview of Follow-up and Validation 
Section 2 - Collecting Follow-up Data 
Section 3 - Reporting Follow-up Results 

Section 4 - Assuring the Accuracy and Validity of Follow-up Data (Validation) 

Each section is divided into subsections according to critical tasks involved in the 
follow-up and validation process. Requirements are presented at the beginning of each 
subsection, where applicable, followed by reconunended strategies for meeting the 
requirements and enhancing accuracy and validity. Illustrative tables, figures, and 
samples appear as exhibits within the text. Additional resource materials appear as 
appendices. 

Once it has been read from beginning to end, the TAG is intended to be a 
reference guide. Certain sections of it will be more useful to those individuals who are 
responsible for the corresponding follow-up tasks. Use the section and subsection 
headings to determine which sections are most appropriate for your needs. For example, 

• Everyone involved in the follow-up process should be familiar with 
the Overview in Section 1, 

• Staff responsible for collecting the follow-up data-particularly 
interviewers and their supervisors-should focus on Section 2, 
Collecting Follow-up Data. 

• Those involved in data analysis and reporting as well as staff 
involved in management and program evaluation should refer to 
Section 3, Reporting Follow-up Results. 

• All persons involved in collecting, processing, reporting, and supervising 
follow-up activities should refer to Section 4 on Validation. 
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1.2 Federal Requirements for Collecting and Reporting Follow-up Data 

Performance standards for PY 1988 and beyond include three postprogram 
measures, each of which reflects a different aspect of the labor market experiences of 
former participants during the 13 weeks following termination from a program. States 
and SDAs (or SSAs in the case of EDWAA) are required to report these measures 
separately for Title U-A adults, Title II-A adult welfare recipients and dislocated 
workers. No follow-up information is required for Title II-A youth. 

Required Follow-up Data 

The follow-up measures include: 

1. Employment rate at follow-up 

This measure is defined as the percentage of total respondents 
(those with completed interviews) who were employed full-time or 
part-time during the 13th full calendar week after termination: 

Number of respondents employed at follow-up 
1 1 X 100 

Total number of respondents 

2. Average weekly earnings (of employed respondents) at follow-up - Title II-A 

For Title II-A this measure is defined as the sum total of gross 
(before tax, health and other deductions) weekly earnings of 
employed respondents during the 13th follow-up week, divided by 
the total number of respondents employed during the 13th full 
calendar week after termination. Average weekly earnings is based 
upon only those respondents employed during the 13th follow-up 
week: 

Total gross weekly earnings of 
respondents employed during the 13th week 



Total number of respondents employed during 
the 13th week 
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Averagi hourly wage (of employed respondents) at follow-up - EDWAA 

For EDWAA, this measure is defined as the sum of gross (before 
tax, health and other deductions) hourly wage rates of employed 
respondents during the 13th follow-up week, divided by the total 
number of respondents employed during the 13th full calendar week 
after termination. Average hourly wage is based upon only those 
respondents employed during the 13th follow-up week: 

Sum of gross hourly wage rates of 
respondents employed during the 13th week 



Total number of respondents employed during 
the 13th week 

3. Average number of weeks worked in the follow-up period 

This measure is defined as the sum of the number of weeks worked 
during the full 13-week period by all respondents, divided by the 
total number of respondents. This results in an average number of 
weeks worked in the follow-up period for all respondents regardless 
of whether they worked at all during the 13-week postprogram 
period: 

Total number of weeks worked in the 
13-week follow-up period for all respondents 



Total number of respondents 

In addition to these measures, the overall response rate and the sample size must 
be reported on the JASR/WAPR for adults, adult welfare recipients, and dislocated 
workers. These items are being used to develop i statistically sound, nationally 
representative database to support valid postprogram standards and adjustment 
procedures. 

Instructions for completing the PY 1989 JASR and WAPR appear respectively in 
Appendices A and B. (Follow-up information appears in Section III of both reports.) 
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Required Response Rates 



When information from follow-up interviews is used for making critical decisions, 
the validity of the information must be assessed. One criterion for assessment is the 
response rate. The response rate is the percen. that actually responded completely to 
the questionnaire out of the total number of possible respondents. (An interview is 
complete only if the respondent answers all of the required questions.) The lower the 
response rate, the lower the degree of confidence in the follow-up data. Having 
confidence that SDAs are performing well or poorly is critical when States reward or 
sanction SDAs. By establishing minimum required response rates, the DOL ensures that 
data are adequate for statistical modeling and that the States can be confident of 
meeting the DOL guidelines for postprogram standards. 

Minimum response rates of 70 percent must be obtained for each of the following 
six groups: 

• Adults Employed at Termination 

• Adults Not Employed at Termination 

• Adult Welfare Recipients Employed at Termination 

• Adult Welfare Recipients Not Employed at Termination 

• Dislocated Workers Employed at Termination 

• Dislocated Workers Not Employed at Termination 

These requirements apply to each SDA/SSA for Title II-A and EDWAA, and to each 
State still reporting follow-up data for Title III. 

Nonresponse bias is a problem that develops when ceitain groups are 
systematically over- or under-represented in the completed follow-up interviews. 
Differences in response rates between respondents who were employed at the time of 
termination and those who were not employed may bias the data: People who are 
employed at termination are more likely to be working at 13 weeks; consequently, they 
may be easier to locate, have higher earnings, and work more weeks in the 13-week 
period than people who were not employed. If people employed at termination are 
over-represented in the completed follow-up interviews, it is more likely that the follow- 
up data will be biased toward higher employment rates and higher earnings. 

For this reason, maximizing response rates for all groups of terminees is 
important to all States and SDAs involved in follow-up. Special attention should be 
given to successfully contacting and interviewing terminees who were not employed at 
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termination. Where the response rates between these two groups are substantially 
different, adjustments for nonresponse bias must be made. Complete instructions for 
making these adjustments are in Section 3.1, Calculating Follow-up Statistics. 

Additional, Optional Follow-up Information 

The follow-up information described above are the minimum data that must be 
collected and reported to DOL to support the development and implementation of 
postprogram performance standards. In order to increase the value of the follow-up 
information for program planning, management, and evaluation. States and SDAs should 
carefully consider collecting additional information beyond the required data. 

In considering the collection of additional information, keep in mind that most of 
the time and expense required to collect follow-up information stems from locating and 
contacting terminees, not from conducting the interviews. Given the fixed cost of simply 
reaching the respondents successfully, it can be very efficient and cost effective to gather 
additional information. 

Additional information might include any or all of the following: 

• AUitudes and opinions regarding JTPA training: A critical area to 
consider is collecting information regarding respondents' perceptions of the 
quality and value of the training received, their need for additional 
services, and suggestions for program changes. Asking such questions is 
useful for monitoring and evaluating programs and service providers, 
identifying problems, providing helpful feedback to service providers, and 
establishing immediate rapport with follow-up respondents. 

Examples: How and why did you enroll in JTPA? 

Was JTPA useful in getting or keeping a job? 

How would you rate the program? 

What was the best thing about the program? worst? 

What one thing would you do to improve the program? 

• Employment history since lea\ing JTPA: Collecting information on 
terminee employment status and activity throughout the 13-week 
follow-up period can provide information that will also be useful in 
evaluating programs and making critical management decisions. 
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Movements into and out of the labor force can be tracked, as can 
shifts between jobs and between employment and additional 
education and training. Detailed information can be collected about 
particular jobs, including hours, wages, benefits, occupation, industry, 
name and location of the firm, and opportunities for advancement. 
Information on EDWAA/Title HI . ^nnincc industry and occupation 
is particularly important since the goal is to re-employ workers in 
new industries and occupations. - interviewers can gather this 
information through a series of structured questions, using the 
reference week, a calendar, and time cues to jog the respondent's 
memory. 

Examples: If currently employed: What is your job title? hours worked? 
fringe benefits? medical benefits? 
If not employed: Are you currently looking for a job? If yes, how 
are you looking? If no, what is the reason you are not 
looking? 

• Receipt of other benefits. In keeping with JTPA's goal to reduce 
welfare dependency, collecting information on the respondent's 
welfare status (e.g., type of welfare, benefit amount) may be 
considered. Consider also collecting data on the respondent's 
receipt of unemployment insurance and disability benefits. 

Example: Are you currently receiving any form of public assistance? 
If yes, how much are you receiving? from whom? 

Appendix C presents an example of one State's questionnaire. 



Additional, Optional Follow-up Data Collection 

In addition to collecting supplementary information from respondents. States and 
SDAs may consider collecting other data that can be useful for broader management and 
evaluation needs. Specifically, data for additional groups of termlnees can provide 
information on a wider range of JTPA participants than Title II-A aduhs and dislocated 
workers. Data on additional periods besides the 13 weeks after termination can provide 
valuable information on longer-term retention and earnings. 
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• Groups of terminees. In addition to collecting infonnation for the 
required groups (Title II-A adults and adult welfare recipients and 
EDWAA/Title III dislocated workers), States and SDAs can collect 
data for other groups: Title II-A youth, Title II-A 3%/8% special 
projects. Title II-B summer youth, and Title IV-C veterans. If large 
enough numbers are involved in the program, you can also select a 
sample of a group, selected by service provider and/or program. 

• Reporting Periods. In addition to collecting information for the 
prescribed 13-week follow-up period, States and SDAs may consider 
collecting data at additional intervals~for example, 26 or 52 weeks 
after termination. Some States conduct 26-week and/or 52-week 
follow-ups to gather information on longer-term retention and wage 
growth. The nature of programs for dislocated workers (EDWAA/ 
Title III) especially suggests that substantial change in earnings, 
occupation, and industry might only occur over a longer period of 
time than 13 weeks. Some SDAs also conduct interim 30- or 60-day 
follow-up to maintain contact with all terminees prior to the 
required follow-up or to gather information needed for local vendor 
monitoring. 

• Data Sources. In addition to the use of participant interviews. 
States may consider using employer surveys and/or administrative 
record data to supplement their postprogram data collection efforts. 
These additional sources, of course, cannot be used for JASR/ 
WAPR reporting in lieu of information gathered directly from JTPA 
terminees. 

Summary of Requirements and Options 

Exhibit 1.1 summarizes the DOL follow-up requirements and options discussed 
above as well as additional requirements described in subsequent sections. 
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Exhibit 1.1 



JTPA Follow-up Data Collection: POL Requirements and Options 



Requirements 


Options 


WHO 

1. All II-A Adult Terminees* 

9 All TT-A AHnIt Wplfarp Terminees* 

3. All EDWAA Terminees* 

*or sample if planned number is more ihan 137, 


1. n-A youth terminees 
? 3% 8% terminees 

3. n-B summer youth 

4. Selected target groups (out of school youth, 
offenders, etc.) 


WHEN 

Data collected following l3th full 

Mi^fiV aftpr tprminatinn and before 

18th full week after termination 

Data recorded on JASR/WAPR 
within 45 days after end of 
Program Year 


1. 26 weeks after termination 
2. 12 months after termination 


WHAT 

1. Employment rate at follow-up 

9 Averape weeklv earninizs fll-A^ of 
employed respondents only 
(average hourly wage for EDWAA) 

3. Average number of weeks worked 
ai follow-up for all respondents 


1. Attitude/opinions 

2. Labor force status 

3. Employment information 

4. Dependency status 


1. Questionnaire by phone 
or in person 

2. Mail, if no telephone 


For 26 week or 12 month follow-up, 
mail or employer contact could also 
be used. 
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13 Using Follow-up Information for Program Evaluation and 

Management 



The purpose of collecting accurate JTPA follow-up data is much broader than 
meeting Federal reporting requirements. Follow-up systems should be designed as an 
integral part of JTPA program management at the local, State, and Federal levels. 
However, because JTPA postprogram standards and the requisite follow-up process are 
relatively new components of JTPA, States and SDAs focused during PY 1986-88 on 
designing and implementing follow-up data collection systems to meet the minimum 
reporting requirements. Although States and SDAs acknowledge the value of follow-up 
data, they sometimes lack the energy and resources required to use these data to their 
full potential. 

One of the most challenging aspects of postprogram follow-up is using the follow- 
up as a tool for reviewing and improving JTPA programs and performance standards at 
the Federal, State, and SDA levels. The relatively recent introduction of follow-up 
standards reflects JTPA's increasing emphasis on long-term employability. As such, 
making follow-up an integral part of JTPA program evaluation and management by 
providing accurate and timely follow-up information is critically important. 



Uses of the Follow-up Data 

Using follow-up information to evaluate and improve JTPA programs at the State 
and local levels, and as a source for developing national standards, has many implications 
for decisionmakers and managers at all levels. Follow-up data may present evidence of 
JTPA's success to government officials and lawmakers, representatives from business and 
industry including PIC members, and the public at large, and thereby can strengthen 
JTPA's credibility. In addition, officials at the Federal, State and local levels can consider 
follow-up data for key planning decisions related to overall service mix and client 
targeting. By linking demographic and program information with follow-up data, they 
can compare the effectiveness of various program activities from a longer-term 
perspective. 

At the State level, in addition to using follow-up data in the granting of incentive 
awards to SDAs, follow-up information can be used to develop and encourage statewide 
coordination of programs. Favorable follow-up results can demonstrate the credibility 
and effectiveness of JITA and thereby facilitate the establishment of statewide 
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collaborative efforts. States may also find follow-up data useful in determining where 
technical assistance is needed and in evaluating the effectiveness of State-level program 
initiativeS"for example, comparing the pre- and postprogram wages and employment 
status of EDWAA/Title III dislocated workers to determine how effective the JTPA 
program was. 

Follow-up data can play a crit.ical role at the local level also. By assessing the 
employment status and earnings of participants 13 weeks after termination, SDAs can 
improve program monitoring and contract reviews. In particular, follow-up information 
can be used to assess individual service providers more comprehensively, compare the 
effectiveness of various program activities, and evaluate the quality of services provided 
to specific target groups. SDAs could integrate follow-up criteria into performance- 
based contracts to provide assurance of results that go beyond enrollments, completions, 
and simple placements. In addition, SDAs can use follow-up data to evaluate their 
internal client services. For example, if the data suggest poor retention after OJT 
placements, an SDA might examine the extent to which inadequately prepared clients 
are referred to OJT or whether the OJT employers are more interested in temporary 
training subsidies than in training permanent employees. By improving accountability for 
longer-term employability, SDAs can significantly improve their job training and 
placement services. 

At the local level also, follow-up information should be used by PICs to 
periodically review the bottom-line results of JTPA programs in relation to expected 
performance. Follow-up standards provide the PIC with a measure of results and the 
beginning of an answer to the important question: Are local JTPA programs increasing 
long-term employability and earnings of their participants? Follow-up information can 
also be used to evaluate the quality of the jobs and benefits provided by employers, 
particularly if additional information regarding benefits (e.g., medical and paid leave) is 
collected. 

In summary, follow-up data, linked with appropriate information such as selected 
demographic variables, training programs, and service providers, can be an invaluable 
tool to: 

• Monitor overall JTPA effectiveness in terms of long-term 
employability goals 

• Predict welfare reduction and other long-term impacts 
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Evaluate and compare the effectiveness of different JTPA programs and 
service providers 



• Identify problems in programs and service providers that require 
further monitoring or corrective actions 

• Assess follow-up results in terms of labor market conditions 
(average wage, unemployment rate, population density, number of 
families below the poverty level) 

• Analyze follow-up results in terms of important demographic factors 
(sex, age, ethnicity, welfare status) 

• Compare current follow-up results with previous results to identify 
trends, identify problems, and recommend corrective actions 

• Compare follow-up results with the performance of similar SDAs or 
States 

• Identify and provide outreach to clients who require further services 

• Provide vital information on program effectiveness to PICs and 
State Job Training Councils responsible for planning and oversight 

• Provide the basis for technical assistance to States, SDAs, and 
follow-up contractors 



Importance of Communicating Follow-up Results 

Depending on the level of decisionmaking, the follow-up data can be used on an 
individual SDA-basis only or aggregated by States or the Federal Government to provide 
broader indicators of program effectiveness. In order for the data to be of any use, it is 
essential that JTPA staff at all levels be informed about the follow-up results in a timely 
manner. For example, in order for SDA staff to monitor, evaluate, and improve local 
JTPA programming, they must have prompt access to follow-up information, whether the 
data are collected at the local or State level. Similarly, for management decisions, like 
establishing incentive policies, that are made at the State level, follow-up information 
must be made available to State decisionmakers. Finally, for management decisions, like 
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setting perfonnance standards, that are made at the Federal level, the follow-up 
information must be forwarded to and used by DOL. 

Involving All JTPA StafT in the Follow-up Process 

Involving JTPA staff at all levels in the initial planning and design of follow-up is 
highly recommended. In particular. States and SDAs should consider the benefits of 
collecting additional information, beyond the required follow-up data, that will help 
program evaluation, reporting, and decisionmaking at both levels. Involving everyone in 
the planning, implementation, and utilization of follow-ups is highly motivational as well: 
everyone is more likely to "buy into" and support follow-up as a valuable part of JTPA 
programming and evaluation, and will be more committed to producing accurate and 
valid follow-up information. Communicating the results of the follow-up to all involved 
is especially critical to reinforcing its importance and usefulness to JTPA staff at all 
levels. 



Limitations in the Usefulness of the Data 

Despite the value of using follow-up information, decisionmakers need to 
understand the limitations of even completely accurate information obtained on all 
terminees in an SDA. The meaningfulness of the data depends on size of each group 
being examined. For example, a small SDA that wants to evaluate differer/. programs in 
terms of their follow-up employment rates might not be able to make reliable 
conclusions based on small numbers of terminees from each program. A more detailed 
discussion of the limitations in interpreting follow-up data appears in Section 3.2, along 
with guidelines for using the data appropriately. 
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1.4 Organizing the Follow-up Intemewing 

Organizing the follow-up interviewing requires joint decisions by States and SDAs 
on two issues: (1) whether the interviewing should be centralized Statewide or 
decentralized (conducted by SDAs), and (2) whether the interviewing should be 
conducted in-house or by an outside follow-up contractor. These decisions have 
implications for both States and SDAs in terms of quality control, staffing, financial 
resources, and fair distribution of incentives across SDAs. 

Since the beginning of required follow-ups in October 1986, States and SDAs have 
adopted and sometimes changed their follow-up systems to best meet their individual 
management needs and their available resources. The majority of States have 
centralized follow-ups as indicated in Exhibit 1.2, which presents the types of follow-up 
systems in place during PY 1987 and 1988: 

Exhibit 1.2 



Number of States* and Types of Follow-up Systems 

PY 1987 PY 1988 

Centralized Follow-up 

State - In-house 12 H 

State - Outside contractor 24** 26** 

Decentralized Follow-up (SDAs) 16 15 



'includes District of Columbia and Puerto Rico 

"includes one State which L centralized except for one SDA 



Regardless of the model adopted. States, SDAs, and outside contractors, where 
involved, must continually communicate and cooperate in implementing, evaluating, and 
modifying the follow-up system to assure accurate and valid results. In evaluating your 
system, consider the effectiveness of your current follow-up in terms of the advantages 
and disadvantages discussed below. 
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Centralized vs. Decentralized Follow-up 

Beginning with PY 1990, all States will grant incentive awards to SDAs based to 
some degree on postprogram performance. The benefits of centralized follow-up are 
related closely to the equitable distribution of these awards. First, a centralized system, 
whether operated in-house or contracted out, has better monitoring and quality control. 
This stems from the administrative efficiency of operating a single, centralized effort and 
the resultant consistency of follow-up procedures implemented across all SDAs in the 
State. 

Second, by centralizing the effort. States can more easily ensure that the data 
have been collected and reported in an objective, consistent, and comparable way. 
When data are collected by persons directly involved in the program being studied (i.e., 
by SDA staff), they may be subject to criticism as being biased. In addition, uniformity 
of procedures enhances the utility of the information by allowing comparisons to be 
made across target groups, programs, and geographic areas. States can grant incentive 
awards based upon follow-up data, confident that the monies have been awarded 
equitably among SDAs across the State. Providing the same assurance for a 
decentrahzed process may require extensive State monitoring. 

Third, a centralized system can be more efficient and cost-effective, particularly 
where there is a statewide MIS capability. Centralization allows administrative 
economies of scale to be realized through lower fixed or negotiated costs. Centralization 
also diminishes the duplication of effort needed to monitor, validate, and provide 
technical assistance for follow-ups in each of the SDAs throughout the State. The need 
for monitoring and validation is discussed in greater detail in Section 4 of the TAG. 

The advantages of a decentralized system in which SDAs conduct follow-up, 
however, should not be overlooked. First, SDAs in control of their own follow-up may 
be able to produce higher response rates because of greater familiarity with their 
terminees and more thorough and timely access to information needed to track and 
locate terminees. Second, SDA staffs are better able to stay in touch with terminees, 
provide outreach services, and be aware of the longer-term outcomes of their programs. 
Third, SDAs that "own" the follow-up may be more motivated to use the data for local 
planning, management, and evaluation needs; can customize their follow-up to better 
meet these needs; and have immediate access to the data. Conversely, they often lack 
the resources and training to accomplish comprehensive follow-up. 
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The decision to centralize or decentralize the follow-up effort requires the close 
involvement of both State and SDA-level staffs in defining mutual roles and 
responsibilities and in cooperatively implementing the follow-up process. 

In-house vs. Contracted Follow-up 

Whether centralized or decentralized, follow-ups can either be conducted by 
in-house staff or contracted out to private or public survey research organizations. The 
advantages to keeping the follow-up in-house are that the time and effort involved in the 
procurement process are avoided and the additional costs involved in an outside contract 
are not incurred. Monitoring the follow-up process is easier, and the risk of hiring a 
poor contractor is eliminated. With in-house interviewing, however, existing staff may be 
burdened with the additional responsibilities of follow-up or extra staff must be found 
and trained to do the interviewing. The work load may fluctuate, requiring changing 
levels of staffing. Staff may be unable to work during evening and off-hours when most 
terminees can be reached. 

Hiring an experienced contractor has several ad ventages. A carefully selected 
contractor can implement a follow-up system efficiently, with minimal startup problems 
and minimal disruption to ongoing State or SDA operations. A qualified contractor 
should have the survey research facilities and capabilities to conduct quality, low-cost 
interviews and to produce high response rates. For example, contractor staff are 
prepared to do interiewing evenings and weekends, while State or SDA staff may be 
reluctant to work such schedules. A contractor is highly motivated to produce high 
response rates in order to retain the follow-up contract. Finally, a contractor can 
enhance the public perception of follow-up as objective and impartial, and thereby 
improve the credibility of the data since the contractor has no vested interest in the 
follow-up results. 

If you decide to hire a contractor, be sure to assess the organization's capabilities 
carefully. Look for these characteristics in a contractor: 

• previous experience conducting interviews with programs and 
participants similar to those in JTPA-preferably familiarity and 
previous experience with JTPA follow-up and with economically 
disadvantaged populations 

• track record in achieving high response rates in telephone surveys 
with similar clientele 
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• adequate telephone interviewing facilities, including multiple phone 
lines 

• an incoming, toll-free 800-line 

• a compatible MIS system ihat can receive, process, and transmit 
data to the State or SDA agency 

• adequate staff and staff training (including, if necessary, bilingual 
interviewers for terminees with limited English ability) 

• reasonable cost per completed interview 

In issuing a Request for Proposal (RFP) for prospective contractors, you should 
include the essential requirements for all anticipated work including any monitoring and 
verification efforts required, the frequency and format of reports, and any option for 
contract renewal. Also include specifications for minimum levels of performance, such 
as response rates, or penalties for missing deadlines and for errors or omissions. The 
RFP can be relatively brief and to the point; an example appears in Appendix D. 

General Guidelines to Promote Accurate and Valid Follow-up Data 

Whether your follow-up is centralized or decentralized, conducted in-house or by 
an outside contractor, it is important to motivate everyone involved in the follow-up 
process to produce accurate, valid follow-up data and maximum response rates. Each 
group involved in the follow-up (State, SDA, and/or contractor) depends on the other 
for various parts of the process: sampling, locating and contacting respondents, 
interviewing, reporting, and using the data. The following practices should promote 
accurate and valid follow-up data: 

• Establish clear and specific procedures for meeting follow-up 
requirements, and distribute these procedures to all involved in 
follow-up. 

• Assign specific roles and responsibilities to each person involved in 
the follow-up-at the State, SDA, and/or ijontractor level. 
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Devise a system to provide current, accurate contact information to the JTPA 
agency or contractor conducting follow-up. 



Publicize, promote, and communicate the importance of follow-up 
throughout the JTPA system-to clients, service providers, 
eiTiploycrs, follow-up contractors, and SDA and State staff. 

Assess the follow-up system you have been using to determine what 
has worked well and what has not; modify your system appropriately 
to improve results. 

Consider using incentives and/or sanctions, making sure that the incentives 
do not themselves introduce bias or temptation to misrepresent results. 
States and SDAs have effectively used incentives such as these: 

- Minimum response rates required to receive full State 
incentives for postprogram measures and/or contractor 
payments 

- Contractor payments based on number of completed 
interviews 

- Interviewers rewarded for completed interviews 

Rapidly communicate the follow-up results to all involved in the 
process to encourage use of the data for planning and evaluation 
purposes and to reinforce future follow-up efforts. 
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Section 2 - Collecting Follow-up Data 



This section presents the requirements and recommendations for collecting follow- 
up data, including sampling (selecting terminees for follow-up); timing (defining the 
follow-up period, scheduling interviews and reporting); locating and contacting 
participants; conducting interviews; training and motivating interviewers; and setting up a 
management information system (MIS) for follow-up. 

2.1 Sampling 

To meet Federal requirements, follow-up information must be collected and 
reported by SDAs/SSAs for three groups: Title II-A adult terminees. Title II-A adult 
welfare terminees, and EDWAA terminees. Spates must collect and report follow-up 
information for EDWAA terminees in statewide, regional, or industrywide projects and 
for EDWAA terminees in National Reserve funded projects. 

If the number of terminees in each group is large enough in a single program 
year, a State or SDA may collect data on a sample of terminees rather than on all 
terminees eligible for follow-up. If done correctly, sampling can provide aggregate, 
accurate estimates of postprogram performance, without the cost of contacting all 
terminees in a group. 

Note that sampled individuals must NOT be selected until alter they have 
terminated their JTPA programs-and NOT upon or during their enrollment. 



Requirements 

When an SDA or State wishes to interview a sample rather than all terminees, it 
must meet these requirements: 

1. For each SDA, separate samples must be drawn for Title II-A adult 
terminees and adult welfare terminees. A sample of dislocated 
worker (EDWAA) terminees is required for each SSA. Separate 
statewide samples must be drawn for two groups of EDWAA 
terminees: (1) terminees in statewide, regional, or industrywide 
projects funded under Section 302(c)(1)(B), and (2) terminees in 
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projects funded under Secretary's National Reserve Grants, funded 
under Section 302(a)(2). 

2. No participant in the follow-up group may be arbitrarily excluded 
from the sample. Therefore, it is critical that all terminee records 
be promptly entered in the database used for sampling. 

3. The process of selecting terminees for each sample must be random 

and must conform to generally accepted statistical practice. 

4. Each sample must be at least as large as either the minimum 
sample sizes or sampling percentages listed in Exhibit 2.1. As you 

can see, sampling cannot be used if fewer than 138 participants in a 
group are expected to terminate over the 12-monih period. In such 
cases, attempts must be made to contact all terminees in the group. 

5. The sampling percentage must be the same for all terminees for 
each group and for each periodic sample from the follow-up cohort 

(participants who terminated during the first three quarters of the 
program year and the last quarter of the previous program year). 

These sampling requirements provide States with the assurance that observed 
postprogram differences among SDAs are, in fact, real differences. This is particularly 
important in the case of Title II-A ^^rograms where SDAs receive incentives or sanctions 
based on postprogram performance measures. The requirements help to ensure: 

• Lack of bias - Random selection from the entire follow-up group 
with uniform sampling rates ensures unbiased estimates of 
performance that are unaffected by seasonal fluctuations or 
concentrations of some types of terminees. 

• High precision of the performance estimates in each SDA. The 
minimum sample sizes ensure that the estimated follow-up 
performance measures across SDAs are precise. There is only a 5% 
chance that an SDA with a follow-up employment rate five 
percentage points below its standard would have met its standard if 
all terminees had been included in the sample. 
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Exhibit 2.1 



Minimum Sample Sizes for Follow-up 



Annual Number of Terminees 
in Population Eligible for 
Follow-up 


Minimum Sample 
Size 


Sampling 
Percentace 


1 - 137 


All 


100 


138 - 149 


137 


94 


150 - 159 


143 


92 


160 - 169 


149 


89 


170 - 179 


154 


87 


180 - 189 


159 


85 


190 - 199 


164 


84 


200 - 224 


175 


82 


225 - 249 


185 


78 


250 - 274 


194 


74 


275 - 299 


202 


71 


300 - 349 


217 


67 


350 - 399 


229 


62 


400 - 449 


240 


57 


450 - 499 


250 


53 


500 - 599 


265 


50 


600 - 749 


282 


44 


750 - 999 


302 


38 


1,000 - 1,499 


325 


30 


1,500 - 1,999 


338 


22 


2,000 - 2,999 


352 


17 


3,000 - 4,999 


364 


12 


5,000 or more 


383 


7.3 
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• Consistency - Uniform sampling procedures improve the 
consistency of follow-up data across SDAs. 

Sampling Decisions 

The Federal requirements only establish minimum sample sizes or sampling 
percentages. Although surveying additional terminees may increase costs somewhat, 
States and SDAs should consider selecting samples larger than the minimum for the 
following reasons: 

• Surveying all terminees may be easier and more cost efTective than 
sampling for small States and SDAs, and even for larger States and 
SDAs that want to analyze differences among subgroups in the 
sample. For example, in an SDA with 150 terminees, the minimum 
sample size is 143. Following up all 150 terminees is easier than 
using a sampling strategy to eliminate seven people. Similarly, a 
large State might find it more meaningful and more cost effective to 
survey all terminees than to administer a weekly, stratified sample. 

• Program managers may want to compare and evaluate the follow- 
up performance of particular programs, service providers, and 
population subgroups (e.g., age, race, gender). These more detailed 
analyses require larger samples for each SDA than the required 
minimum sample sizes, which are designed to provide accurate, 
aggregate estimates of the performance of SDAs, SSAs, or States 
(for statewide, regional, and industrywide projects or Secretary's 
National Reserve Grants under EDWAA). 

• Oversampling can provide protection against fluctuations in the 
number of terminees during the year and elimination of terminees 
from the sample due to death, institutionalization (e.g., in hospitals. 



2.1 Sampling - 26 



prisons, nursing homes), or severe incapacitation that makes them 
unable to be interviewed for the entire follow-up period. 

• Surveying all terminees simplifies the procedures used to analyze the 
follow-up data since a weighting procedure must be used to 
aggregate SDA results into statewide totals unless each SDA uses 
the same samphng percentage. 

Sampling Procedures 

Preparing for sampling. The following steps must be taken prior to selecting the 
sample: 

• Define a regular time period for sampling: it can be weekly, 
biweekly, or monthly (recommended for most SDAs). 

• Make sure all terminations for the period are entered In the MIS by 
the time you plan to select the sample-preferably less than 9 weeks 
after the beginning of the period. Otherwise, you have broken a 
cardinal rule of random sampling, which requires that all terminees 
must have an equal chance of being selected for the sample. If the 
terminee record is not in the database, it has been arbitrarily 
excluded. 

• Develop lists of all adult terminees in the period. The lists must 
identify welfare status and ideally would contain primary and backup 
contact (at least two) information for each terminee. Generally, the 
SDA would prepare this list, though the State could do it through a 
centralized or networked MIS. The list should be prepared no later 
than 10 weeks after the beginning of the follow-up period. 

• Send the lists of terminees, ideally in machine readable form, to the 
group responsible for sampling (State, SDA, or contractor as 
appropriate) no later than 10 weeks after the beginning of the 
termination period. 
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Determining the minimum sample size or sampling percentage. The minimum 
sample size and sampling percentage were both designed to meet the same statistical 
criterion and differ only because of the use of ranges and rounding. States or SDAs may 
choose to use either method. For ease of explanation, "minimum sample size" is used 
below. 

The first step in sampling is to determine the required minimum sample size by 
locating the planned annual number of terminees for each group in the left-hand column 
of Exhibit 2.1. For each group of terminees eligible for follow-up, a corresponding 
minimum sample size is needed to estimate SDA-level follow-up performance for that 
group. The required minimum sample size appears in the middle column. The right- 
hand column expresses the minimum sample size a' a percent of the number of planned 
terminees. Either the minimum sample size or the sampling percentage can be used to 
determine a sample size that will meet the DOL requirement. For example, suppose an 
SDA estimates it will serve 550 adult II-A terminees in the Program Year. Exhibit 2.1 
shows that a minimum sample size of 265 or a sampling percentage of 50% is required 
for 550 terminees. The SDA must therefore select at least 50% of adult II-A terminees 
for each sampling p'^ . iod throughout the year. 

Once a sampling percentage is determined, it must be applied throughout the 
whole year. Changing the sampling percentage during the year to correct for inaccurate 
estimates can bias the results of the follow-up. If the original number of planned 
terminees is too high, you cannot adjust the sampling proportion to meet the minimum 
sampling requirement. Thus, using a somewhat larger than minimum sampling 
percentage is recommended to assure meeting the sample size requirement. 

Sampling to reflect planned number of terminees accurately. Terminees must be 
sampled periodically (monthly or weekly) and consistently so that the total sample 
accurately reflects the flow of terminees from programs. The percentage of the size of 
the periodic sample to the size of the group of terminees eligible for follow-up each 
period should equal the percentage of ♦he total sample size to the size of the total group 
eligible for follow-up. For example, if an SDA selects a sample size of 50% of all 
expected adult terminees for the year, it should draw a 50% sample each sampling 
period. In a period that had 150 terminees, the sample would be approximately 75. In a 
period with only 80 terminees, the sample would be about 40. The samples from all 
periods combined will equal the total sample size for the year, or 50% of all terminees. 

The entire samphng scheme depends on the accuracy of SDA plans because 
sampling proportions are based upon planned, or expected, numbers of terminees. 
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Exhibit 2.1 shows that the lower the number of terminees, the greater the percentage of 
terminees that should be sampled. Therefore, if actual terminations run behind plan, too 
few terminees will be included in the sample and the required minimum sample size may 
not be met. Conversely, underestimated planning figures can lead to oversampling and 
to higher than necessary sampling costs. 

The following procedures are provided to help States or SDAs obtain adequate 
and unbiased samples that reflect follow-up outcomes throughout the 12-month period. 
(These procedures were adapted from the State of Indiana's Sampling System, revised in 
April 1989.) 

1. To determine the total number of terminees expected to be served 
by an SDA, review the performance standards plan submitted by the 
SDA. Note the number of planned terminations for the upcoming 
program year. 

2. To verify these planning figures, review data for the past two years 
to analyze trends. First, compare the planning figures with the total 
actual number of terminees in each gvoup for each of the past two 
years. If the planning figures look re;ilistic compared to the past 
years' data, they can be used as the basis for determining the 
appropriate sampling percentage. Then, compare the sample sizes 
to the total terminations for the past two years to make sure that 
the sampling percentages used in the past were adequate and are 
appropriate for the upcoming year. 

3. Once you determine the "realistic" number of terminations for the 
upcoming year, use Exhibit 2.1 to establish the appropriate sampling 
percentage, as described earlier. 

4. Determine the number to include in each sample by multiplying the 
number of terminations falling within the appropriate range of dates 
by the sampling percentage. Round the result to the next whole 
number to provide a small, automatic cushion against 
undersampling. 

Using random sampling. The process for selecting terminees for inclusion in the 
sample must be random. That is, all terminees in each group must have an equal chance 
of being selected. No participant may be arbitrarily or systematically excluded from the 
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sample: individuals employed at termination and those not employed at termination must 
all have an equal chance of being selected. Any method consistent with accepted 
statistical procedures m?v be used for selecting the random sample. Several examples of 
random sampling methods appear in Appendix E (How to Select a Random Sample). 

Backup sampling. Individuals who have died or are incapacitated may be 
excluded from the sample and from the calculation of response rates. This includes 
persons who are institutionalized (e.g., in hospitals, prisons, nursing homes) or severely 
incapacitated and unable to be interviewed for the entire follow-up period. If 
eliminating such individuals drops the sample below the minimum size, you need to 
replace them in order to obtain the minimum sample size. Any excluded individual 
should be replaced by another termiriee randomly selected from the same group 
(e.g., II-A adult) and month of termination as the excluded terminee. If, as previously 
recommended, States and SDAs initially draw random samples greater than the 
minimum size required, backup sampling is not necessary. 

Selecting the Title II-A welfare sample. The Title II-A adult welfare sample will 
be selected in two parts. First, some adult welfare terminees will be selected as part of 
the overall Title II-A adult sample. Then, you must randomly draw an additional 
supplemental welfare sample from those welfare recipients that were eligible for 
follow-up, but were not already selected in the adult sample. 

To determine the size of the supplemental welfare sample, determine the 
minimum sample size for the adult welfare recipients. Subtract from this number those 
welfare recipients already included in the total Title II-A adult sample. (The ratio of 
welfare adults to all adults in the sample drawn for follow-up may differ from the ratio 
in the total terminee population for the year. This discrepancy is not important.) The 
result is the minimum size of the supplemental welfare sample. An example follows on 
the next page. 

In many SDAs, the number of adult welfare terminees in a given year will be so 
small that a 100% sample would be necessary. In such cases SDAs would draw the 
overall adult terminee sample, picking up some portion of the welfare terminee 
population in the process, and then include all remaining adult welfare recipients in the 
welfare follow-up sample. 
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An SDA expects to terminate 1,600 adults in Title II-A during Program Year 
1989. Of these, 640 will be welfare recipients. Welfare recipients will 
represent 40% [(640/1,600) x 100] of the adult terminees. 

The required minimum sample sizes, taken from Exhibit 2.1 are: 
Minimum adult sample size = 338 
Minimum welfare sample size = 282 

If randomly drawn, approximately 40% of the adult sample could be expected 
to be welfare recipients. Forty percent of the adult sample of 338 equals 135. 
So the adult welfare supplemental sample would be 147, because: 

282 adult welfare sample size 
- recipients already in the a dult sample 
147 supplemental welfare sample 

The supplemental welfare sample of 147 individuals must be selected from the 
505 welfare recipients not previously selected for the sample (640 - 135 = 505). 
This represents a sampling percentage of 29% (147 divided by 505). 
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Requirements 

Three requirements are involved in timing the follow-up: the duration of the 
follow-up period; the time span, or window during which interviewing may occur; and the 
particular groups, or cohorts of terminees who arc followed up and reported each 
quarter. 

1. The postprogram, or follow-up period for all terminees is the 13 w^ekg 
following termination from a JTPA program. For a given terminee, the follow-up period 
begins on the first Sunday following the termination date and concludes on the 14th 
Saturday following the date of termination. The last full calendar week of the follow-up 
period, from the 13th Sunday to the 14th Saturday, is designated as the reference week. 
The data reported in the JASR/WAPR should reflect the employment rate and average 
weekly earnings during the reference week, and average number of weeks worked during 
the entire 13-week follow-up period. (The data reported on the WAPR should report 
the average hourly wage, rather than the weekly earnings.) 

2. The time span, or window for data collection is limited to a 4-week period 
from the 14th through the 17th calendar week following termination. Postprogram data 
collection may extend beyond this 4-week window through the 18th week only when the 
individual has been located but not interviewed before the end of the 17th week or when 
the individual is temporarily absent during the regular 4-week period for data collection. 

3. Follow-up data must be collected and reported to DOL for individuals who 
terminated from the program in the first three quarters of the current program year, as 
well as those who terminated from the program in the last quarter of the previous 
program year. (This group is called the follow-up cohort.) This is because the follow- 
up interviews lag the termination dates by about 13 weeks. The schedule of quarterly 
cohorts for whom follow-up interviews are required in PY 1989 and PY 1990 appears in 
Exhibit 2.2. States and SDAs can schedule and conduct inurviews on a monthly or 
weekly basis, using a program year calendar to identify appropriate reference weeks and 
interviewing windows. The follow-up data must be entered on the JASR/WAPR and 
reported to DOL no later than 45 days following the end of the program year. 
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Exhibit 2.2 

Annual Required Follow-up Cohorts by Termination Quarter 



PY 1988 


PY 1989 


PY 1990 


Q4 


Ql 


Q2 


Q3 


Q4 


Ql 


Q2 


Q3 


Q4 



PY 1989 PY 1990 

Follow-up Cohorts Follow-up Cohorts 



Timing Decisions 

The follow-up interview schedule controls the timing of the entire data collection 
process. Exhibit 2.3 summarizes key activities in the follow-up process and recommends 
a schedule for conducting them in order to meet data collection and reporting 
requirements. 

Consider, in addition, these scheduling suggestions for conducting key follow-up 
activities: 

• Provide lists of terminees with accompanying contact information to 
interviewers no later than 10 or 11 weeks after termination. This 
allows time to prepare and mail notification letters, if used, and to 
schedule interviews. If the lists are provided to interviewers once a 
month, contact information will need to be generated as early as six 
weeks after termination. This process should be coordinated with 
the MIS data entry process. 

• Because the follow-up workload varies with the number of 
terminees, flexible stafling may be needed to accommodate peak 
periods of terminations. 
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When using mail surveys to contact hard-to-reach terminees, all 
attempts at telephone interviews should be made during the 14th 
week to allow enough time to mail questionnaires to terminees so 
that complete responses mailed by the end of the 18th week can be 
included in the data reported on the JASR/WAPR. 

The data collection window leads to a very tight reporting schedule. 
Consider a termination that occurs on the last day of the third 
quarter. The reference week ends on the first Saturday after the 
program year, followed by four weeks to complete the interview. 
Thus, the interview period ends approximately 31 days after the 
close of the program year, leaving only 14 days to enter, edit, and 
report the data to DOL. 
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Exhibit 2.3 



Summaiy of Key Activities 



TIME PERIOD 


TASK 


Prior to and during JTPA 
participation/termination 


• Inform participants of follow-up, and obtain 
their agreement to participate. 

• Collect and update primary and back-up contact 
information for all participants. 

• Promptly enter termination data into the MIS. 


Weeks 10-11 
Post-Termination 


• Prepare list of terminees eligible for follow-up. 

• Select sample(s). 

• Prepare notification letters. 


Week 13 
Post-Termination 


• Prepare questionnaires, as applicable. 

• Mail pre-notification letters. (Send earlier if using 
reply cards.) 


Weeks 14-17 
Post-Termination 


• Contact and interview all terminees included in 
sample. 

• Make callbacks, as needed. 

• Code and enter questionnaire data, as required. 


Weeks 18-19 
Post-Termination 


• Complete remaining interviews and data entry for 
terminees who mail back questionnaires by the end 
01 ine loin weeK. 


End of Program Year 


• Prepare aggregated reports for required groups. 

• Adjust for non-response, as required. 


45 days after 

End of Program Year 


• Submit JASR/WAPRs to DOL. 
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2.3 Locating and Contacting Participants 

States, SDAs, and follow-up contractors have found that JTPA tcrminees rarely 
refuse to provide follow-up data, even relatively sensitive information like their income 
or welfare status. The rate of refusals ranges from zero to less than five percent. 
Therefore, the main challenges in conducting an effective follow-up are in locating and 
contacting terminees. Achieving a 70% response rate requires a comprehensive and 
efficient system to do this. This section presents the Federal requirements for locating 
and contacting terminees and recommended strategies for obtaining maximum response 
rates. Appendix F, Selected Survey Research References, cites several standard survey 
research texts that cover these issues in more detail. 



Requirements 

Only one general requirement must be met in locating and contacting participants 
for follow-up: The techniques used to contact terminees must not introduce bias. The 
same effort should be made to contact all types of terminees regardless of their 
termination status or demographic characteristics. In particular, similar effort must be 
made to locate individuals who were not employed at termination (and are typically 
harder to contact) and those who were employed at termination. 



Obtaining Adequate Contact Information 

The key to effective follow-up data collection is the quality and accessibility of the 
contact information obtained from participants while they are still in the program. The 
following techniques have been used to obtain this information effectively: 

• Upon enrollment, obtain the name, address, and telephone number 
of the participant and at least two backup contacts. These contacts 
should be relatives or friends who do not live with the participant 
but would likely know how to find the participant later on. "In case 
of emergency" contacts are not h.^^A if they are family members at 
the same residence, who may be as difficult to locate as the 
participant. Parents and other older relatives not living with the 
participant are ideal because they are less mobile. 
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• Require service providers to provide primary and multiple baclcup 
contact information on each terminee as part of their performance 
contracts. (This increased the response rate in one SDA by 5-10%.) 

• Periodically during training and upon termination, verify and 
update the contact information. 

- Give a prepaid postcard to participants to return with 
changed contact information. 

- Require participants to provide updated contact information 
on time sheets required for needs-based payments, where 
appropriate, 

- Feed the updated information into the MIS in a timely way. 

- Maintain contact with participants who are in the 90-day 
period of inactive status. Terminate them at the end of this 
period, and enter the data into the MIS promptly. 

• Establish linkages and develop cooperative relationships with other 
agencies and staff that might have contact with participants (SDA 
counselors, local employers, JTPA service providers, social services, 
correction agencies, community service organizations, homeless 
shelters). Encourage interviewers to use these resources after a few 
unsuccessful attempts to contact participants. A thank-you note 
might be sent to individuals who have helped locate terminees. 

• Communicate <*ontact information in a timely manner and make it 
easily accessible to follow-up interviewers-preferably through a MIS 
network. 

Preparing Participants for Follow-up and Motivating Them to Participate 

Terminees will be more willing to provide follow-up information if they know 
about follow-up in advance and understand its importance and legitimacy. The following 
steps are recommended: 
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upon enrollment and termination, inform participants of the 
importance of follow-up and their responsibility to provide follow- 
up information, if requested to do so. 



- Obtain the individual's signed agreement to participate in 
.follov>'-up and to provide contact information, at the time of 
enrollment. A sample Follow-up Agreement appears in 
Exhibit 2.4. 

- Require counselors or service providers to have an exit 
interview with participants to identify any need for additional 
services, to remind them about follow up, and io obtain 
updated contact information. Record the dale and the name 
of the exit interviewer in the MIS for use in establishing 
rapport in the follow-up interview. 

- Obtain the participant's permission to request information 
from employers. This will enable interviewers to call 
employers for help in locating employed terminees and in 
verifying follow-up data provided. An example of a 
statement used to obtain the participant's permission to 
contact employers is included in the Follow-up Agreement in 
Exhibit 2.4. (See Section 4.2 for more information on 
employer verification.) 
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Exhibit 2.4 



Sample Follow-up Agreement 

Participant Name SS# 

Program Operator 

You are being enrolled in a training program funded under the Job Training Partnership 
Act (JTPA). The program is administered by [SDA] and operated by [Service Provider]. 
Its goal is to hel' ♦ you obtain the education or training that will lead to your future 
employment. 

In order to measure how well we are meeting this goal, we need your help. After you 
leave this program, you will be contacted by [Follow-up Agency/Contractor] to see how 
you are doing and to ask you a few questions. By answering these questions, you will 
provide us with information that will help us to serve you and others better in the future. 

One of [Follow-up Agency/Contractor] interviewers will call you 14 weeks after you 
leave the program. It will take only a few minutes, and all the information you provide 
will be kept strictly confldential. Your participation is very important to the future 
success of this program. We appreciate your help. 

List at least two people who do not live in your household and will always know vour 
whereabouts. 



Name Address Phone 

Name Address Phone 



I voluntarily agree to provide information in the follow-up interview. I understand that 
my participation is voluntary and that the information I provide will be kept strictly 
confidential. 

I also hereby give permission to my past/present employer to release information to the 
[Follow-up Agency/Contractor] regarding my employment and earnings. 



Signature of Participant Date 



Signature of Program Date 
Representative 
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• About one week before the expected interview date, mail notification letters 
or post cards to follow-up participants: 



- Identify the purpose of the interview, its voluntary and 
confidential nature, and the specific information that will be 
requested (employment and earnings). The letter might 
include a calendar to highlight the reference weeks. 

- Remind them of previous references to follow-up, if 
appropriate-for example a Follow-up Agreement. 

- Provide a bilingual translation of the letter or post card for 
non-English speaking participants. If possible, these 
individuals should be identified in advance on the MIS. 
Translations of letters should be printed on the reverse side 
of the English letter, and both sides of the letter should 
indicate that the other version is on the reverse side. 

- Include methods to verify the accuracy of available contact 
information and to encourage participant to call in. For 
example, ask participants to provide their current phone 
number on a business-reply post-card or to call in on an 800- 
line or collect. Or subscribe to the U.S. Postal Service's 
Address Correction Service, which will notify you of any new 
address and its effective date, and charge only for successful 
tracking attempts (approximately $.30). 

A sample notification letter appears in Exhibit 2.5. Use or adapt it to meet your 
follow-up procedures. 
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Sample Notification Letter 
[SDA/CONTRACTOR LETTCRHEAD] 
[DATE] 

[PARTICIPANTS NAME] 
[PARTICIPANT'S ADDRESS] 

Dear [Mr., Ms.]: 

The [SDA/CONTRACTOR NAME] is interested in learning more about you and other 
persons who have recently participated in a Job Training Partnership Act (JTPA) 
program. While you were in the [NAME OF PROGRAM/VENDOR] several months 
ago, you may recall being told that you might be asked to participate in a follow-up 
survey after you left the program. 

Someone from our office will be calling you in a few days to ask you some questions 
about your employment experiences during the past several months. Information you 
provide will be used in evaluating the programs and will be held in strictest confidence. 
Your participation is voluntary and your answers will not affect your eligibility for 
AFDC, food stamps, unemployment insurance, or similar programs. 

Questions will include: 

1. Did you do any work for pay during the week beginning [START/END 
DATES]? 

2. If you worked during that week, how many hours did you work that 
week, how much did you get paid per hour, and how much extra, if 
any, did you earn in that week from tips, overtime, bonuses, 
commissions, or any work you did on the side before deductions. 

3. How many weeks did you work at all for pay during the entire 13- 
week period from [DATE] through [DATE]? 

A calendar is included with this letter to help you answer the questions. 

We will be calling you at [PARTICIPANTS PHONE NUMBER], the telephone number 
that is in our records. If you have a new telephone number, please write or call [NAME 
OF INTERVIEWER] at [PHONE NUMBER] so that we can get in touch with you. 

Those of us here at [SDA/CONTRACTOR] look forward to talking with you soon. 
What we learn from you will help us to improve the programs we provide. 
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Exhibit 2.5 (Continued) 



1989 



JANUARY 1989 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 
15 16 17 18 19 20 21 
22 23 24 25 26 27 28 
29 30 31 



FEBRUARY 1989 
S M T W T |- S 
12 3 4 
5 6 7 8 9 10 11 

12 13 14 15 16 17 18 

19 20 21 22 23 2; 25 

26 27 28 



MARCH 1989 
S W, T W T F S 
12 3 4 
5 6 7 8 9 10 11 
12 13 14 15 16 17 18 
19 20 21 22 23 24 25 
26 27 28 29 30 31 
30 



APRIL 1989 
S M T W T F S 
1 

2 3 4 5 6 7 8 
9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 



MAY 1989 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



SEPTEMBER 1989 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 
10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 



JUNE 1989 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 



OCTOBER 1989 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 31 



JULY 1989 
S M T W T F S 
1 

2 3 4 5 6 7 8 
9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 31 

NOVEMBER 1989 
S M T W T F S 
12 3 4 
5 6 7 8 9 10 11 

12 13 14 15 16 17 18 

19 20 21 22 23 24 25 

26 27 28 29 30 



AUGUST 1989 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



DECEMBER 1989 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 
10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 
31 



1990 



JANUARY 1990 
S M T W T F S 
1 2 3 4 5 6 
7 8 9 10 11 12 13 
14 15 16 17 18 19 20 
21 22 23 24 25 26 27 
28 29 30 31 



FEBRUARY 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 

11 12 13 14 15 16 17 

18 19 20 21 22 23 24 

25 26 27 28 



MARCH 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 31 



APRIL 1990 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 



MAY 1990 

5 M T W T F S 

1 2 3 4 5 

6 7 8 9 10 11 12 
13 14 15 16 17 18 19 
20 21 22 23 24 25 26 
27 28 29 30 31 



JUNE 1990 
S M T W T F S 
1 2 

3 4 5 6 7 8 9 

10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 27 28 29 30 



JULY 1990 
S M T W T F S 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 

15 16 17 18 19 20 21 

22 23 24 25 26 27 28 

29 30 31 



AUGUST 1990 
S M T W T F S 
12 3 4 
5 6 7 8 9 10 11 
12 13 14 15 16 17 18 
19 20 21 22 23 24 25 
26 27 28 29 30 31 



SEPTEMBER 
S M T W T 



'.990 
F S 
1 

2 3 4 5 6 7 8 

9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 



OCTOBER 1990 
S M T W T F S 
1 2 3 4 5 6 
7 8 9 10 11 12 13 

14 15 16 17 18 19 20 

21 22 23 24 25 26 27 

28 29 30 31 



NOVEMBER 1990 
S M T W T F S 
1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 26 27 28 29 30 



DECEMBER 1990 
S M T W T F S 
1 

2 3 4 5 6 7 8 

9 10 11 12 13 14 15 
16 17 18 19 20 21 22 
23 24 25 26 27 28 29 
30 31 
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Contacting Participants for Follow-up 



Obtaining maximum response rates requires systematic and common sense 
procedures for contacting participants. After conducting follow-ups for several years, 
States and SDAs have found the following strategies effective: 

• Time your calls to reach participants successAiUy. Analyze your 
own contact records to determine the best times to call. In general: 

- Avoid calling during standard business hours when 
employed terminces are at work. Interviewers have found 
the best times to call are early evening hours (5 - 8 Monday 
through Friday), Saturday 10 - 2, Sunday evenings, holidays. 
Bad weather days are also good times to call. 

- Vary times for repeated calls if a participant is hard to reach. 

- If you reach an answering machine or someone other than the participant, 
leave a message for the participant to call back. 

In States or SDAs where calling during off hours is difficult for staff, 
consider offering compensatory time or monetary incentives to staff, 
rotating after-hours assignments among staff, or hiring special 
follow-up staff who can work flexible schedules. 

• Once you have reached participants, interview them immediately 
whenever possible. If completing the interview is not convenient at 
that time, arrange to call again at a more convenient time or for the 
person to call back easily: 

- Establish a toll-free 800-number if possible (it's easier and 
less intimidating to participants than calling collect). Forty 
percent of one State's response rate comes from callbacks on 
an 800-number, and several States have reported 
improvements in response rates of more than 20% from 
800-number callbacks. 
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- Whoever answers the phone must be able to conduct the 
interview or immediately transfer the call to someone who 
can. Assign a distinctive, nctitious name to the interviewer 
(that is neither male nor female, like Chris Jasser) so that 
return follow-up calls can be recognized and referred 
immediately to appropriate staff. 

• Establish systematic procedures for reach* terminees that include: 

- As many telephone attempts as necessaiy to reach the 
terminee: often six or more attempts are needed to reach 
participants. 

- a prescribed order for attempting contacts such as the following: 

1. Home phone 

2. Backup contacts 

3. SDA program staff 

4. Service provider (vendor) that trained participant 

5. Directory assistance 

6. Other directories such as local telephone books or cross- 
referenced/reverse directories that are published by private firms. 
Calls to other r ^.mbers listed under the same name or at the same 
address may provide information useful in locating a terminee. 

7. Employer or worksite of job-placed terminees 

Employers may provide useful information for locating terminees. 
By contacting the employer, it may be possible to speak with the 
participant to arrange an interview, leave a number for the 
participant to call, or get information on how to locate the 
participant. When calling employers, interviewers should be careful 
not to release confidential information about the participant, not to 
disrupt the participant's work day, and not to irritate an employer 
who could be the source of future placements. If possible, 
interviewers should establish a contact person at the employer, and 
call no more than once or twice in a week. Note that you cannot 
use any information about employment status or earnings obtained 
from employers for JASR/WAPR reporting purposes In Ueu of 
information obtained directly from JTPA terminees. 
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• a required record, or log, of all attempted contacts. The log 

should include the date, time, and method of each attempted 
contact; the name or ID number of the staff member making 
contact; and the outcome of each contact. It should be 
available for follow-up supervisors at all levels to use for 
verification and monitoring purposes. See Exhibit 2.6 for a 
sample Contact Log Form. 

• a follow-up mailing to hard-to-reach terminees who have an 
active address on record. Mailing should be a last resort 
only, as it has not been a very effective initial follow-up 
strategy. (See Section 2.4 for more information on mailing 
questionnaires.) 

• Consider assigning the most hard-to-reach cases to a single 

interviewer or to a small number of especially qualified interviewers 
during third or fourth follow-up week. Although not practical for 
small SDAs, this strategy can improve the efficiency of a large 
follow-up system. 
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Exhibit 2.6 



Sample Contact Log Form 



COMPLETED J _ 
SDA: 
MAME: 
ADDRESS: 

PHONE NO: 
ALT. NOa 

WORK NO: 
NEW NO; K 



WEEK mt 

QUEST. »t 
SOCIAL SECURITY NO: 

TERM DATE: mtti/dd/yy 

ACTIVITY: 

TEF^M TYPE: h 



VENDOR: 
EMPLOYER: 
DATE: /. 



WA&E: »9i.'3'5 



TITLE: 

IIA= 
IIAU» 
111 = 
YOUTH= 
YOUTH WELF= 



DATES INTERVIEWED: / /. 

/ / 



CALL RECORD 



CALL 


DATE 


TIME AM«0/PM-l CALL CODE DISPOSITION 


CONTACT INrORMATlCf'4 




1 


/ 


/ 










/ 


1 




















3 












4 


_/ 


J 








5 __. 


_/' 


J 








6 __. 


_/__ 


J 








7 


J 








8 


/ 


/ 


X 






■1 


/ 


/ 


• 






(J /_. 


J 


t 







14 15-16/17-18/19-20 21-24 2S 
CODES; 

l>CAlltd horrid 

1-CalUd kln/fri*nd 

2»Calltd Directory Assist. 

4«Calltd *tripl oy^rf 

S*C«lUd S^rvicr D*liv<fry Arr« 

bsCalUd training faciltiy 

7-Kart;cipant >;all«rd in 



NOTE: PUNCH OfiLY LAST -INE 



2& 27 
DISPOSITION CODES: 

0>No answer /busy signal (IP> ^ 

l«Particlpant not avallabU. R*sid*nc* Confir'«i«d IP* 

2«Int#rvi#w Partially Coi^pUtifd (IP) 

3«Phont # Disconn#ct»d/Chang#d/Not in S*rvio*/Nwt L.ived 
4«Participant no longer at i/Providifd no info:<iation 
5«Provid#d n#w afldrMs/phone i <1PJ 
6»Institutionali2*d/unabU to int#rvi*w (F) 
7«P*tusal on scr^*ning/t*niiinat#d during inttrr/.rw r 
e>Unabl# to contact (F) 
>Coi«pUt#d Int»rvi#w (F) 
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2.4 Conducting Follow-up Interviews 



Requirements 

To fulfill the follow-up reporting requirements, all of the following information 
inust be collected from respondents:' (1) whether the respondent was employed during 
the 13th week after termination (the reference week); (2) if employed, respondent's 
hours worked, hourly wage, and additional earned income in the 13th week; and (3) the 
total number of weeks the respondent worked during the 13-week period ending with the 
reference week. The following requirements for conducting follow-up interviews must be 
met: / 

• Interview questions developed by DOL must be used to determine 
the follow-up information reported on the JASR and WAPR. 

Participants must be told that responding is voluntary and that 
information they provide will be kept confidential. The required 
DOL questions appear in Exhibit 2.7. 

• Attitudinal questions may be asked before the required DOL 
questions. However, the required questions must be asked before 
any other questions about employment or earnings. 

• The required information must be gathered directly from the 
participant. Although relatives, friends, and employers may provide 
contact information, the questions must be asked directly of the 
participant. 

• The information should be obtained by telephone or in person. 
Mail questionnaires may be used in those cases where an Individual 
cannot be reached by telephone or in person. Telephone interviews 
should be the primary method for collecting follow-up data. In- 
person interviews may be appropriate when respondent? drop into 
JTPA offices, in rural areas without telephones, and for specific 
hard-to-reach terminees such as the homeless. Mail surveys should 
be a last resort only. More information regarding mail 
questionnaires appears later in this section. 
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Minimum Postprogram Data Collection Questions 

A. I want to ask you about the week starting on Sunday, , and ending on Saturday, , 

which was (last week/two/three/four weeks ago). 

1. Did you do any work for pay during that week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour in that week? 

Dollars per hour 

4. How much extra, if any, did you earn in that week from tips, overtime, bonuses, commissions, or 
any work you did on the side, before deductions? 

DoUars 

B. Now I want to ask you about the entire 13 weeks from Sunday, , to Saturday, . 

5. Including the week we just talked about, how many weeks did you work at all for pay during the 
13-week period? 

Weeks [Go to end] 

ALTERNATIVE QUESTIONS 
C If answered "NO"" to Question 1: 

Now I want to ask you about the entire 13 weeks from Sunday, , to Saturday, , 

6. Did you do any work for pay during that 13-weck period? 

Yes [Go to 7] 

No [Go to end] 

7. How many weeks did you do any work at all for pay during that 13-wcek period? 

Weeks 
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Interviewing Participants to Obtain Complete and Accurate Information 

Establishing rapport. Once contacted, participants usually agree to provide 
follow-up information. However, they sometimes do not recognize "JTPA" or are 
reluctant to provide personal information such a' wages. Therefore it is important to 
establish rapport, trust, and cooperation immediately. Interviewers should introduce 
themselves and be friendly and informal. Tliey should briefly explain the importance of 
follow-up, its use for program improvement, and its voluntary and confidential nature. 
To convey familiarity with participants and concern about their JTPA experience, 
interviewers should: 

• Identify the participant's specific training program and service 
provider. (This information should be available through the MIS.) 

• If a notification letter was sent and to establish your credibility 
immediately, ask the participant if the letter was received. 

• Begin interviewing as soon as possible. The most efficient and 
effective way to convince respondents that you are not trying to sell 
them anything is to ask the interview questions immediately. 

• Ask participants for their opinion about their JTPA training before 
asking the required DOL questions-informally if these questions are 
not part of the actual questionnaire. 

A sample telephone introduction appears in Exhibit 2.8 at the end of this section. 

Clarifying questions and probing. Clarifying questions or probing is necessary 
when a respondent does not understand a question or when a response is incomplete, 
unclear, or probably inaccurate. Probing should be neutral: it should not lead the 
respondent to give a particular answer. Explanations, or clarifications, of the required 
questions appear in Exhibit 2.9, and examples of acceptable probes appear in 
Exhibit 2.10. 

Confronting language problems. If your State or SDA serves a substantial 
number of non-English speaking people, hiring or identifying a local source of bilingual 
interviewers is important. If language is an occasional problem, arrange for another 
person in the participant's home to translate, scheduling a callback if needed. However, 
make sure that the participant provides the information, not the interpreter. 
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Converting potential refusals. Once contacted, reluctant participants may be 
persuaded to complete the follow-up questions; similarly, reluctant friends or relatives 
may be encouraged to locate the participant. Reasons for possible refusals and 
suggestions for gaining cooperation appear in Exhibit 2.11. 

Using a mailed questionnaire. As mentioned earlier, use mail questionnaires only 
for terminees who cannot be reached by phone or in person. The cover letter should 
refer to the initial notification letter (if there was one), restating the purpose of the 
interview and explaining its confidential nature. It should also provide a return 
telephone number (preferably an 800-line) and hours when return calls can be made by 
respondents. Alternately, the letter might include a business reply postcard requesting a 
current name, address, and telephone number, and a convenient time for the interviewer 
to call. The mailed questionnaire should be sent in enough time to reach terminees by 
the end of the 17th week, so that complete responses mailed by the end of the 18th week 
can be included in the data on the JASR/WAPR. 

Experience of States and SDAs indicates that terminees are more likely to call 
back to respond than to return a mailed questionnaire, particularly a lengthy one. To 
increase the likelihood of a response, the mailed questionnaire should be brief and easy 
to read, fill out, and return. Include the required questions only, worded appropriately 
for a mailing, and a calendar for reference. Including optioaal questions in the mail 
questionnaire is not recomip'^nded. Be sure to include a self-addressed, stamped 
envelope. 

An example of a cover letter and mail questionnaire appears as Exhibit 2.12. 
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Sample Telephone Interview Introduction 

Hello, my name is [ ], and I am calling from the [Name of 

State, PIC, SDA, Contractor as appropriate]. As you know from the letter we sent you 
recently, we are conducting a brief survey of the [JTPA Training Program and Service 
Provider] in which you participated three or four months ago. 

I would like to ask you a few questions about what you have been doing since you 
left the program. All of your answers will be kept strictly confidential and are voluntary 
on your part. Your responses will help us to improve employment and training programs 
in the future. The questions are very short, so it will only take a few minutes. 

[Note to Interviewer: Do not sound hesitant or tentative. This may be the only time the 
participant will be reached. Push for completion, and at the very least, get answers to 
the required questions. If the participant cannot be interviewed immediately, schedule a 
later time to conduct the interview, or ask the participant to call back on the 800- 
number or collect.] 
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Exhibit 2.9 



Clarifications of Reguired Questions 



Required DOL Questions 


Clarifications 


Question Al: 

Did you do any work for pay 
during that week? 

Yes 

No 


The answer to this question should be "yes" if at 
any time during the 7 consecutive days of the 
reference week the respondent was: 

• a paid employee; 

• a member of the armed forces on active duty or 
a participant in a registered apprenticeship 
program; 

• self-employed in his/her own business, 
profession or farm; or 

• absent from work or the business due to illness, 
bad weather, vacation or personal reasons, but 
still on paid status. 


Question A3: 

How much did you get paid 
per hour on that week? 

$ hour 


The following items are included in hourly wages: 

• all wages, including pay for vacation and 
sick leave. 

• amounts which may have been deducted from 
the paycheck for taxes, health insurance, credit 
unions, retirement, child support, alimony, etc. 

• nouriy wages irom an jods neia aurmg reierence 
week. 

The following items are not included in hourly 
wages: 

• money received through pensions or transfer 
payments such as welfare, social security, or 
unemployment insurance. 


Question B5 and C7: 

Now I want to ask you about 
the entire 13 weeks starting 
from Sunday, 
to Saturdav. 

[including the week we just 
talked about]. How many 
weeks did you work at all for 
payduring the 13-week period? 

Weeks 


This question asks the respondent to mentally 
calculate the total number of weeks worked during 
this 13-week period. Only whether the respondent 
has worked at any time in a given week is 
important, not the amount of time worked during 
the week. Again, individuals on paid sick leave 
or paid vacation should be counted as working. 

The interviewer may clarify the status of each of 
each of the 13 weeks with the help of a calendar, 
using the categories provided under question Al. 
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Acceptable Interview Probes 



Types of Probes 


Examples 


Repeat or rephrase 
a question 


^ere you paid for any work you did 
last week? 


Repeat or rephrase the 
respondent's answer 


You said you were paid $7.50 per hour, but I did 
not hear how many hours you worked last week. 


Make neutral statements 
requesting more information 


I'm not sure I understand. Could you explain further. 


Specify key events or dates 
to identify the reference period 
or suggest the respondent 
look at a calendar 


The 13th week was the week of Halloween. 


Probe or ask back-up 
questions to obtain earnings 
information-use available 
termination information to 
jog their memory. 


When you left the program, you were hired full-time 
at Acme Body Shop at $6.65 per hour. Were you 
making the same amount last week? Did you work 
40 hours last week? 


Clear up an apparent 
discrepancy error in a response 


I thought you said you were not working last week 
(the 13th week) but that you worked for the entire 
13 weeks since [date]. Did you really work all 13 
weeks or did you mean that you were working every 
week except last week? 
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Converting Refusals 



Reasons for Refusal 


Response 


Too busy/no time 


I only have a few questions, but would be happy to 
call you back at a more convenient time. When 
would be a good time for me to call back in the next 
day or two? 


Bad health 


I'm sorry to hear that. I would be happy to call back 
in a day or two. Would that be okay? 


Feels inadequate/confused 


The questions are not difficult. They concern how 
you feel about the job training program. Some of 
the people we have interviewed had the same concern 
you have, but once we got started they didn't have 
any trouble answering the questions. Would you like 
me to read a few questions to you and you can see 
what they're like? 


Questions too personal 


Your answers will be completely confidential. No 
one else will know how much you were paid. 
Protecting people's privacy is one of our main 
concerns. We add up the answers from lots of people 
to find out how well the program is working. 


Not interested/sees no benefit 


It's very important that we get the opinions of 
everyone who participated in the program. Your 
opinions and experiences with [program] will help 
us learn how useful the program was to you, so it 
will be useful for others. 


Objects to telephone surveys 


We have had a lot of experience and success with 
teleohone survevs like this one flf Darticioant is 
Still reluctant...] However, if it would, be convenient 
for you, I would be happy to mail a questionnaire to 
your home. 
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Exhibit 2.12 



Sample Mail Letter and Questionnaire 
[SDA/CONTRACTOR LETTERHEAD] 



[DATE] 

[PARTICIPANT'S NAME] 
[PARTICIPANT'S ADDRESS] 



Dear [Mr., Ms.]: 



Several months ago you participated in a job training program funded by the Job 
Training Partnership Act (JTPA). Our records indicate that you were enrolled in 
[NAME OF PROGRAM/VENDOR]. We are now conducting a follow-up survey to 
find out how successful the program was in helping you. You may recall being told 
about this survey while you were in the program. Your cooperation would be greatly 
appreciated. 

We have been unable to reach you by telephone, so we are sending you a brief 
questionnaire. Please take a few minutes to fill out jilJ of the items on the questionnaire, 
and return it to us in the enclosed stamped envelope. It is important that we receive 
your response before [End-date of 4 .wggk interview "window"], so answering quickly 
would be appreciated. If it would be more convenient, feel free to call [interviewer 
name] us with your answers at [800-number and/or other number] between [appropriate 
hours/days]. 

Your responses are voluntary and will be held in the strictest confidence. The 
answers you give will in no way affect your eligibility for this or other, similar programs. 

Thank you for your help and let us know if we can be of assistance to you in the 

future. 

Sincerely, 

Program Director /Survey Manager 

Enclosures 
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Exhibit 2.12 (continued) 

Sample Mail Questionnaire 

We would like to know about your work experiences during the 13-week period 

shown on the calendar below-starting on Sunday, , and ending on 

Saturday, . 

[Insert small calendar with appropriate weeks highlighted.] 

1. Did you do any work for pay during the 13th week, starting on Sunday, , 

and ending Saturday, ? 

Yes 

No [If NO, go to question 2.] 

la. How many hours did you work in that week? 
Hours worked 

lb. How much did you get paid per hour in that week? 
Dollars per hour 

Ic. How much extra, if any, did you earn in that week from tips, overtime, 
bonuses, commissions, or any work you did on the side, before deductions? 
Dollars 

2. How many weeks did you work at all for pay during the entire 13-week period, 
starting on Sunday, , and ending Saturday, ? 

Weeks worked 

Thank you very much. Please return this questionnaire in the self-addressed, stamped 
envelope as soon as possible. 
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2^ Training and Motivating Interviewers 



Training, supervising, and motivating interviewers are essential components of an 
effective follow-up system, whether the foliow-up is conducted statewide or at the SDA 
level, in-house or by a contractor. Systematic training is important to familiarize staff 
with JTPA and the importance of follow-up, with the procedural requirements of follow- 
up, and with strategies for obtaining accurate, valid data and high response rates. 

Training, Supervising, and Evaluating Interviewers 

Sample interviewer training materials appear in Exhibits 2.13, 2.14, and 2.15. 
These materials have been adapted from materials developed by Employment and 
Training Institute, ETI, a private follow-up contractor in Ringwood, New Jersey. Exhibit 
2.1? provides a suggested interviewer training outhne; details for most of the topics in 
the outline are provided in the TAG sections indicated on the outline. The Orientation 
Guide in Exhibit 2.14 provides information for Section I of the outline relating to the 
background and purpose of JTPA and follow-up data collection. An example of a form 
that can be used to evaluate interviewers appears in Exhibit 2.15. 

Providing interviewers with sufficient training and supervision can ensure that they 
understand the purposes and procedures involved in follow-up, can deal with hard-to- 
locate and reluctant participants, and will collect and record follow-up information 
completely and accurately. For example, new intei-viewers who have not previously 
talked with economically disadvantaged people may have difficulty remaining objective 
and not be tempted to offer personally to help. Helping interviewers develop techniques 
for remaining objective, while establishing rapport, would be valuaole. 

In addition to training and regularly monitoring the work of interviewers, the supervisor 
should meet regularly with interviewers to provide feedback, to discuss problems and 
effective ways to address them, and to share each other's effective strategies. 



Motivating Interviewers 

In addition to training, supervising, and providing feedback to interviewers, 
consider motivating interviewers in the following ways: 
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• Establish friendly, informal competition among staff by periodically 
reporting results. For example, post results for each interviewer in 
terms of completed interviews per day or most challenging locations 
reached. One State uses a wall map to plot the locations of faraway 
respondents interviewed. 

• Acknowledge or reward interviewers for achieving high response 
rates or using innovative strategies. For example, designate an 
interviewer of the month, and hold a monthly "ceremony" to 
acknowledge successful staff. Engage interview staff in the process 
of determining appr'^priate incentives. 

• Pay interviewers or contractors for completed interviews. Consider 
paying bonuses for high response rates. (One State pays its follow- 
up contractor a bonus for each percentage point over a 70% 
response rate.) However, if incentives are used, monitoring 
interviewers is especially critical to ensure honest and thorough 
performance. 

• Improve staff accountability by integrating interviewer's 
performance on follow-ups into the performance appraisal process. 
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Training Outline for Interviewers 

I. Project Background and Purpose [Exhibit 2.14] 

A. The Job Training Partnership Act (JTPA) 

1. Funding 

2. Explanation of Private Industry Council (PIC) 

3. Explanation of Service Deliveiy Area (SDA) 

4. Discussion of participant eligibility for Title II-A and EDWAA 

5. Description of program activities (Classroom Training, OJT, Direct 
Placement) 

6. Description and role of community agencies (service providers, 
employment service, welfare) 

7. Organization of State/ SDA program staff 

B. Follow-up Data Collection 

1. Significance of results in terms of performance standards 
[TAG Sections 1.1 and 1.3] 

2. Use of results for evaluating and monitoring programs and service 
providers [TAG Section 1.3] 

3. Explanation of the 13 week follow-up period, reference week, and the 
"window" for data collection [TAG Section 2.2] 

4. State, SDA, or Contractor's experience with follow-ups [TAG Section 1.4] 

II. Procedures for Locating and Contacting Respondents [TAG Section 2.3] 

III. Review of Interview Questions and Procedures for Recording Responses 
[TAG Section 2.4] 

IV. Interviewing Principles and Procedures [TAG Section 2.4] 

A. Establishing rapport with respondents 

B. Responding to different types of respondents (typical, reluctant) 

C. Pacing the flow of the interview 

D. Probing to clarify questions and responses 

E. Reversing refusals 

V. Role Playing Exercise (mock interviewing) 
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nmriBni's omrmai ooisi 

TO JTPA AD PQ8TPI00IAM POXOH-OP 

As a Postprograa Pollow-Op Inttrvltwtr, you will b* in contact with ptopla who 
havt gona through tha JTPA (Job Training Partnarahlp Act) syataa. This guida 
will attaapt to acquaint you with thia aystaa. 



BACKQSOOID 

JTPA is a fadarally-fundad Jobs prograa gaarad priaarily to aarring tha aco- 
nomically disadvantagad. It has baan In aziatanca ainoa 1982. Pravioualy, tha 
fedaral Jobs prograa was callad CBTA (Coaprahansiva Eaployaant 4 Training Act). 

The Job Training Partnarahlp Act (JTPA), anactad in 1982, rapraasnta an antiraly 
naw concept in providing Job akill training for tha nation's unaaployad diaad- 
vantaged and dislocated workers. Funded in 1984 at $3.6 billion, this Act 
channels federal dollars through the atatea to looal pritata industry councils 
for training progress. Tbeae local councila auat be chairad by business 
persons, and local business leadera auat aake up a aajority of the council 
aeabershlp. 

The local private and public leaders serving on private induatry councils 
deteraine what eaployaent skills are needed in their ooaaunlties. They can then 
use federal funds to establish looal trslnlng prograaa that provide the 
disadvantaged uneaployed with the skilla to aatoh identified local eaployaent 
needs. The councils continually review prograa operations and evaluate actual 
results such as the percentage of traineea eaployed, oost per placeaent, and 
other perforaanca criteria. They then deteraine if new or different akill 
training prograas are needed. 

The Job Training Partnership Act la designed to insure that public funds are 
spent effectively to provide needed Job skills. . Seventy percent of all funds 
Bust be used for training prograaa that lead to private aactor eaployaent. 
Adainistrativa costs are liaited to a aaziaua of 15*. At leaat 40} of the funds 
aust be used to train disadvantaged and uneaployed youth— the nation's aost 
critical category of uneaployaenU (Minority youth uneaployaent ia now -jvar 
40$. Non-ainority youth uneoployaent is over 20% today.) 



THE TRAIVDIG PROGRAM 

Based on the decisions aade by the PIC, public and private training schools are 
chosen (through a bidding process) to train JTPA-eligible participants into Jobs 
(often entry level). The Act providea that a wide variety of training services 
aay be nade available to youth and adults with prograa funds. These services 
Include, but are not liaited to: 

* Job Factory or Job Search Assistance — A group of activitiea which 
provide labor aarket inforaation, prograaa for individuals and groups, 
Job clubs, and individualized Job search plans, . This assistance can be 
beneficial and inexpenaive (with very liaited supportive services 
provided) for clients with a aarketable Job akill. Job aearoh assistance 
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Exhibit 2.14 (continued) 



alao Bay b% a follov-up aotitity for olitota vbo bayt aueeaaafullx 
coapltttd a akilla tralnlot prograB. 

* Diract Placaiant ~ Tha prograa raorulta allfflbla clianta aod r%f%n thai 
to an axiating Job opaDlng. Vary llttla oounaallag or tralnltif la 
providad. 

* Pra-7ooational Trainlog — Gaoarally a ^aaarooa, atruoturad actlTlty 
vbleb providaa labor aarkat InforaatioD, introduotion to tba vorld-of- 
vork, inatructlon on oorract eoaplatiofi of raauaaa and aaploysant 
applicationa, and an avaranaaa of raquirad vorkplaoa babayior. 

a or Bl-Lingual Education ~ Spaciallaad inatruotion for non-Engliab 
and lialtad-Engliab apaaking clianta daaignad to aaka ttaaa aufficlantly 
fuoctional in Engllab to procaad vitb tbair iodlTldual aaployabllity 
plan* 

* CRT - Ranadial Education and Baaic Skllla Training Qanarally pro- 
grajBsad instruction to laprova raading, writing and ooaputational akilla 
of clianta who bava aducational dafielanclaa. (Tbia la oftan callad 
••Pre-Vocatlonal Training.") 

* High School Equivalancy (GED) Ganarally an actlTlty, oouplad with 
raaedial aducation, which la daaignad to proTlda an aqulTalancy cartifi- 
cate to individuala lacking a high acbool diplosa. 

* On-tha*Job Training (OJT) — A Job plaoaaant actlTlty which proTldaa for 
raiaburaaaent to tba amployar of an aaount aqual to aa auob aa 50$ of tha 
antry-leval aalary of an aaployaa, OJT ooapanaataa tba aaployar for 
axtraordinary auparviaion and Batarlal coata and raducad productlTlty 
during tha naw aaployaa*a training pariod. OJT prograaa ara gana^^ally 
viavad aa tha aoat affactlTa Job plaoaaant toola bacauaa tba aaployar la 
ablt to acraan and to aalact tba oaw aaployaaa and than train that, 
according to that aaployar* a apacific naada. 

* CRT - Occupational — Traditional ocoupation-'apaclfio training, which la 
providad by a public or privata training Tandor with dasonatratad 
training capability. SDAa aay buy daaa-alsa training or purcbaat 
indlTldual training alota, or Bay nagotiata with tba Tandor to cuatoaiza 
a apacial prograa to train for an Idantlflad daaand akill. 



SPECIAL TOOTB PROGRAMS 

Beyond tha aarvicaa daacribad ao far, tba local SDA Bay cbooaa to fund 
additional prograjna to aaat tha apacial aaployaant naada of youth in the 
labor aarkat araa. Tba Act daacribaa four auob prograBai any ona or all of 
which may ba included In tba Job training plan# 

* Pre-Eaployaant Skllla Training Prograa An in-*acbool prograa for youth, 
including 14 and 15 year olda, daaignad to provide up to 250 boura of 
inatruction and actlvitiaa in a variety of labor aarkati Job aearchi and 
baalc aklll areaa. Priority abould ba glTen to tboaa not aeeting 
eatibllabed levela of acadaaic aebieTeBent who plan to enter the full- 
tiae labor aarkat upon leaving acbool. 
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Exhibit 2.14 (continued) 



Entry Eaployntnt Szptrltnot Program — A progru proTldlog lubsldliwl 
•iployaeDt opportunltlM In tbt publlo and prlTatt aactors for 
Indlvldutla vho ara anrollad In saooadary lattl tduoatlon, vltb priority 
to tboaa not planning to go on to poat^saooodary •duoatloni and vho matt 
otber rastrlctlona. 

Scbool- to-Work Tranaltlon Asslatanct Prograa i prograa dlractad at 
blgb school atniors planning to antar tha full*tlaa labor atrkat upon 
graduation. Transition sanricas aay inoluda Job saarch tasistanot and 
rsfsrral to training if nacasaaryi Including raftrral to adult training 
activitits and tha Job Corps. 



FUIDIIQ 

JTPA funding is al3ocattd by "subgroups" for tha purposs of ssrving diffarent 
segments of a community's eligible population. Subgroup funding sources are 
referred to as "titles". For examplet Title IXA funds programs which serve the 
economically disadvantaged. This category includes adulta» youth and veterans. 
Title IIA receives the majority of funding through JTPA. Title III monies fund 
programs for dislocated workers, according to a percentage of the tliglble 
population. 



THE PABTICIFAIT 

This section will, hopefully, give you an idea of the process a participant goes 
through in order to receive JTPA services. Although the JTPA process varies 
from state to state and SDA to SDA, the following Is a general overview. 

Many participants come into the JTPA system through looal Employment Services 
(Unemployment Offices, Jot Service); others through City aad County Welfare 
(AFDC^ WIN); some are recruited by individual training programs (Vendors), and 
many others through word-of-mouth referrals (friends or relatives who have gone 
through a JTPA program). 

In order for an individual to find out if he la eligible for a JTPA program, be 
must bring documentation of Income, age, address, family alse, etc Baaed on 
economic need (or other eligibility), the peraon la "certified" aa eligible. He 
usually is then tested academically (and, in some cttes, for aptitude). Each 
training program has its own entrance requirementa which must be met before 
referral. The individual then meets with a counselor to discuss his goalSi 
abilitiesi and test results. Based on that interview, a referral is made. The 
participant may be referred to classroom training, OJT, special programs, or a 
social service agency. Supportive services such aa transportation and/or child 
care allowance aay also be made available. 

Many JTPA participants have social, educational, economic and psychological 
barriers to overcome in order to complete a training program and/or obtain 
employment. For these reasons, many drop out of programs. 
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Exhibit 2.14 (continued) 



Tou will bt inttrvltvlog partldpanti vbo coapltttd trtlnlnf (rtftrr^d to u 
*po8ltltt"} and aont who did oot ooBplttt training (rafarrad to ai 'non-poai- 
tiva"^). Tha aaaa rafaranoa ia applioabla to OJT partioipanta. JTPA alao uaaa 
codaa to raflaot tba activity of a participant during hia anrollaant, Tou vill 
racaiva a list of UotlTltr Codaa* ao you vill knov if jou ara addra^alng 
aoaaona who baa gona througb claaarooa trainingi OJT| or apaoial prograaa. 

posTPioQUN raxov-op 

This ia tba fourth yaar tbat follow-up ia baing oonductad. Tba Dapartaant of 
Labor baa aaodatad that all JTPA prograaa conduct a follow-up of its partioi- 
panta (this can ba dona by Stata or SDA). 

In tha paati JTPA could only Judga ita affaotiTanaaa and parforaanca in taraa of 
positiva placanants (thosa placad in Joba upon taraination froa prograaa); thara 
has baan no way to assaas tha prograa'a long-tara raaulta. 

Follow-up on aacb participant will ba conductad at 13-vaak intarvala (aftar 
laavlng the prograa). Tba intarviaw will ravaal tha participant's work history 
(or lack of ona) ainca laaving tba prograa, and alao balp ua gain aoaa insight 
into tba participant's axpariancaa with tha JTPA systaa* 

Inforaation gainad froa this study will allow ua to dataraina, in part, waak- 
nassaa which aight axist within a particular JTPA ayataa, poor work hiatory 
aaong cartaln subgroups, and poor ratantion rataa raaulting froa placaaant 
through individual vandors. 



CORCLUSIOi 

Postprograa Follow-Up ia now baing conductad acroaa tba natioa Tba raaulta of 
this atudy will ba analyzed and validated by aganciaa on atata and national 
levels. 

Ab an interviewer, you ara tba aoat critical coaponent of the prograa. Tou 
vill be receiving tha responses froa partioipanta which will aaka thia atudy 
possible. 

The purpose of Postprograa Follow-Op is to aacartain the effectiveness of JTPA 
prograaa over the long tera. Hopefully, this study will produce inforaation 
which will be utilized to iaprova JTPA's overall ability to train a qualified 
and productive workforce* 
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Exhibit 2.15 



Interviewer Evaluation 



Interviewer Date 

Supervisor 

General Intemew Performance 

Veiy Good Amm Needs Work 

• Rapport with respondent (1(1 (1 

• Pace of interview (1(1 (1 
(ability to control questions) 

• Flow of interview (1(1 (1 
(smooth transition) 

• Accuracy in asking questions (1(1 (1 

• Verbal skills (1(1 (1 
(conversational quality, use 

of inflection) 

• Probing I 1 I 1 I 1 
(neutral, secures appropriate 

responses) 

• Good judgment (1(1 (1 
(responds well to challenges) 

• Accuracy in recording information I ] I 1 (1 



Commenis 

Specific comments on interviewer's performance 



Supervisor recommendations 
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2.6 Setting Up a Management Information System (MIS) for Follow-up 



Requirements 

There are no official requirements for Management Information System (MIS) 
procedures related to follow-up data- collection. However, certain data collection and 
information retrieval procedures are essential to an effective follow-up effort. The 
availability and accessibility of up-to-date addresses and telephone numbers of terminees 
and at least two baclcup contacts are the keys to locating most of the follow-up sample. 
A listing of client characteristics will enable interviewers to confirm that they have 
reached the correct respondent, will help them to establish rapport, and will identify 
non-English speaking individuals who may need bilingual interviewers. Many 
respondents may know the type of training they received and their service provider but 
not the SDA name or "JTPA." Therefore, information on the participant's training 
program and service provider, as well as their employment status at termination, 
employer's name, and termination wage, will enable interviewers to establish credibility 
and jog the respondents' memory of their JTPA experiences. 



Making Adequate MIS Information Available 

Whether the MIS system is manual or automated, the following information about 
each terminee in the sample should be retrieved and made available to interviewers: 

• Name 

• SDA 

. JTPA Title (n-A Adult, II-A Adult Welfare, EDWAA) 

• Current telephone number and mailing address 

. Backup telephone number and address of at least one, but preferably two, 
people who can locate a respondent who cannot be reached (not 
"emergency numbers" of relatives who live with the terminee) 

• Age 

• Gender 

• Ethnicity 

• Native language and whether the terminee has limited English 
abilities to match the interviewer's language to that of the terminee 
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• Key dates including date of termination and start and end dates of 
the follow-up period and the reference week 

• Name, provider, location, and type of training received 

• Status at termination (placed or not placed) 

If placed at termination: 

• Placement employer's name, address, and telephone number 

• Wage at placement 

Increasing the Accuracy and Usefulness of the Follow-up Data 

To facilitate follow-up and to increase the accuracy and usefulness of the follow- 
up data, you need to establish procedures for providing adequate MIS information to 
interviewers, for editing and entering the follow-up data into the MIS, and for analyzing 
and reporting the data in a meaningful way. With only five postprogram reporting 
elements on the JASR/WAPR, Federal reports could probably be completed manually, 
particularly for small follow-up efforts. However, processing and meaningful analysis of 
follow-up data, especially of large data files, are best done by computer. Most States, 
SDAs, and contractors today have automated MIS systems that can be used to streamline 
follow-up operations. The following strategies have been used effectively: 

• Provide to interviewers a single computer-generated form for each 
terminee that includes participant data and a record of follow-up 
contacts. The questions can also be included. (An example of such 
a form appears in Exhibit 2.6 in the previous section of the TAG.) 

• Use a machine-readable form to simplify data entry. 

• Establish a Computer-Assisted Telephone Interview (CATI) system 
to eliminate the need for separate data entry and to ensure 
complete and accurate data collection and reporting. (See 
Appendix H for a description of the software to support JTPA 
follow-up.) 

• Build in MIS edits to checlc for internal consistency and 
reasonableness in the follow-up data. Use local labor market 
conditions and conmion sense to establish appropriate limits-for 
example, that an acceptable hourly wage ranges from $3.80 - $15.00, 
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and that hours worked during the 13th week range from 0-70. Use 
related data items to identify discrcpancics-for example, a large 
difference between the wage at placement and follow-up weekly 
earnings. The DOL uses the following edit checks for consistency 
and reasonableness of SDA-level data when processing your 
JASR/WAPR data: 

. For Title II-A terminees, the Follow-up Weekly Earnings are compared to 
the Average Wage at Placement: 

Follow-up Weekly Earnings 
Average Wage at Placement 
The acceptable range is 30 - 55 hours. 

- The Employment Rate at Follow-up is compared to the Average Weeks 
Worked: 

Average Number of Weeks Worked x 100 

+ Follow-up Employment Rate 

13 

Tlie acceptable range is 0.8 to 1.2. 

Implement procedures for timely investigation of suspect data and 
error correction. 

Establish efficient programs for merging participant data with the 
follow-up data in order to produce reports that provide information 
useful for evaluation and decision-making. 
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Section 3 - Reporting FoIIow-up Results 



Once the follow-up data have been collected, the results must be compiled and 
reported to DOL according to their specifications. Certain calculations must be made to 
comply with requirements and to ensure that the data reported accurately represent the 
employment status and wages of eligible respondents during the 13-week follow-up 
period. This section describes the requirements for calculating and reporting response 
rates and for completing the follow-up portions of the JASR and WAPR. It also 
describes guidelines for interpreting and reporting follow-up data for State and local 
evaluation and management purposes. 



3.1 Calculating Follow-up Statistics 

As explained in Section 1.2, minimum response rates of 70% must be obtained for 
each of six group of terminees surveyed. The value of the data is even greater if a 
higher response rate can be obtained. Response rates must be calculated separately for 
Title II-A adults and adult welfare recipients and for EDWAA dislocated workers who 
were employed at termination and those who were not employed at termination. 
Separate calculations are needed in order to determine whether either group is 
underrepresented among respondents. Follow-up statistics should be adjusted for any 
nonresponse bias introduced by differences in response rates. They must be adjusted if 
the difference in response rates between individuals who were employed at termination 
and those who were not is more than five percent (5%). 



Defining Response Rate for Follow-up Data Collection 

The response rate is defined as the number of terminees who provide complete 
follow-up information required by DOL, divided by the total number of terminees 
included in the follow-up sample. It is expressed as the following percentage, rounded to 
tenths (e.g., 73.6%): 

# terminees with complete follow-up interviews 
— X 100 

# terminees selected for follow-up 

The denominator, or base, of the response rate will be the number of terminees 
in the follow-up sample. Even if the State or SDA is following up all eligible terminees 
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or more than the minimum required sample size, the denominator must include every 
terminee thu was selected. 

An interview is considered complete only if the terminee has provided answers to 
all of the required follow-up questions. Responses such as "don't know" or "don't 
remember" to the required questions are not counted as acceptable answers. For 
example, an interview is not counted as comphtc if the terminee provides information 
about employment status and earnings during the 13th follow-up week but cannot 
provide the number of weeks worked during the follow-up period. This terminee would 
be excluded from the numerator, but would be included in the denominator in 
calculating the response rate. 

In calculating the response rate, persons institutionalized (e.g., in hospitals, 
prisons, nursing homes) or those who have died or are severely incapacitated (and 
therefore are unable to be interviewed for the entire follow-up period) should be 
excluded from the sample. I'he number of terminees who fall into these categories is 
generally quite small. All other terminees selected for the follow-up sample must be 
included in the denominator. 



Adjusting Follow-up Statistics for Nonresponse Bias 

As described in Section 1.2, nonresponse bias occurs when certain groups of 
terminees are over- or under-represented among the completed follow-up interviews. 
The follow-up results must accurately reflect all selected terminees, not merely those 
from whom responses could be obtained. Therefore, where the response rates between 
two groups of terminees (e.g., those who were employed at termination and those who 
were not) are substantially different, adjustments for nonresponse bias must be made. 

Adjustments are made by weighting the follow-up data collected according to the 
proportion of individuals employed at termination and the proportion not employed at 
termination. Adjusting for nonresponse bias is a straightforward process which can be 
done manually following the procedures presented in Appendix G. Appendix G 
includes: 

• a worksheet and instructions for calculating differences in response 
rates to determine whether nonresponse bias adjustments are 
required 
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• a worksheet and instructions for computing adjustments for 
nonresponse bias 

Alternately, the adjustments can be done by computer using the "Report" option 
on the software to support JTPA follow-up, which is described in Appendix H. An 
example that illustrates the manual adjustment procedure follows: 

Example of Nonresponse Bias Adjustment 

Suppose that: 

• the group eligible for follow-up includes 200 individuals employed at 
termination and 100 individuals not employed at termination. The 
SDA is following up on a 100% sample. 

• 180 of those employed at termination (90%) completed the 
interview, and 85% of these respondents (153 of them) were 
employed at follow-up. 

• 60 of those not employed at termination (60%) completed the 
interview, and 40% of these respondents (24 of them) were 
employed at follow-up. 

Then, using the procedures presented in Appendix G: 

• The correct (or adjusted) follow-up employment rate would be 70% 
not 73.8%: 

Correct (adjusted) InCQITgtt (P9t a<»iystgd) 

200 X 153 
180 

70% 

300 

170 + 40 
300 

In the correct (adjusted) resuU, each group contributes to the overall 
statistic in proportion to its representation in the total terminee population, 
not in the completed sample. 



100 X 24 



60 



73.8% - 



153 + 24 



180 + 60 
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Failing to adjust for nonresponse bias will proportionately magnity the 
performance of groups with higher response rates and discount the performance of 
groups with lower respo^ise rates. In other words, if those employed at termination have 
higher response rates (as they typically do), failing to make the adjustment produces a 
combined follow-up performance that weights the information more for respondents 
employed at termination than for the respondents not employed at termination. 
Although the adjustment does not correct for all potential nonresponse biases, it does 
diminish the bias. 

States or SDAs are required to adjust for nonresponse bias for all three 
postprogram measures: employment rate at follow-up, average weekly earnings, and 
average weeks worked. An adjustment must also be applied to each of the groups 
requiring follow-up data collection: Title II-A adults. Title II-A adult welfare recipients, 
and EDWAA terminees. 

Presently, the adjustment for nonresponse bias is required only when the 
difference in response rates between those terminees who were employed and those who 
were not employed at termination exceeds five percentage points. If the difference 
between the response rates is five percent (5%) or less, adjusting for nonresponse is 
optional since its effect on follow-up performance may be quite small. Nevertheless, a 
uniform procedure that makes the adjustment in all cases is recommended. 

Calculating the Required Follow-up Data 

Accurately calculating th^. required follow-up data is a critical prerequisite to 
reporting and utilizing valid follow-up information. Instructions for calculating and 
reporting the required follow-up measures appear in detail in the JASR and WAPR 
instructions (Appendices A and B). Instructions are summarized below, in terms of 
responses to the required follow-up questions, presented in Exhibit 2.7 (Q refers to the 
question number, and R, to the total number of respondents): 
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• Employment rate at follow-up: Divide the number of respondents who 
answered '*yes" to Question 1 (employed at follow-up) by the total number of 
respondents (R). Then» multiply the result by 100 to get a percentage. 

Number of respondents employed at follow-up Sum [Ql] 
1 X 100 = X 100 

Total number of respondents R 

• Average weekly earnings (of employed) at follow-up - (Title II-A/JASR): 

Divide the sum of the products of the answers to Question 2 (hours per week) 
and Question 3 (hourly wage) plus the sum of the answers to Question 4 
(other earned income) by the number of respondents who answered "yes" to 
Question 1 (employed at follow-up). 

Total gross weekly earnings of 
respondents employed during the 13th week Sum [(Q2 x Q3) + Q4] 

Total number of respondents employed during Sum [Ql] 

the 13th week 

• Average hourly wage (of employed) at follow-up - (EDWAA/WAPR): Add the 
sum of the answers to Question 3 (hourly wage) to the sum of the answers to 
Question 4 (other earned income) divided by Question 2 (hours per week). 
Divide the result by the number of respondents that answered "yes" to 
Question 1 (employed at follow-up). 

Sum of gross hourly wage rates of 
respondents employed during the 13th week Sura [Q3 + Q4/Q2] 

Total number of respondents employed during Sum [Ql] 

the 13th week 

Note that the average hourly wage/weekly earnings of employed respondents 
at follow-up is based upon only those respondents employed during the 13th 
follow-up week. 



3.1 Calculating Follow-up Statistics - 75 



77 



• Average number of weeks worked in follow-up period: Divide the sum of the 

answers to Question 5 (weeks worked, employed at follow-up) plus the sum of 
the answers to Question 7 (weeks worked, not employed at follow-up) by the 
total number of respondents. 

Total number of weeks worked in the 
13-week follow-up period for all respondents Sum [Q5] + Sum [Q7] 



Total number of respondents R 
the 13th week 

Note that this calculation is based on all respondents, not just those 
employed. It results in an average number of weeks worked in the follow-up 
period for all respondents regardless of whether they worked at all during the 
i3-week postprogram period. 

• Sample sizes Rejjort the total number of terminees selected for contact, 
including both respondents and nonrespondents. Only deceased, 
institutionalized (e.g., in hospitals, prisons, nursing homes), or severely 
incapacitated and unable to be interviewed for the entire follow-up period can 
be excluded from the sample. 

Note: All sums are to be based on answers from all respondents in the 
appropriate group (II-A Adult, II-A Adult Welfare, EDWAA). Exclude from all 
calculations any answers from nonrespondents (i.e., terminees who do not answer all the 
relevant required questions). 
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3.2 - Reporting Follow-up Results 



Reporting Results to the DOL 

Follow-up data collection and reporting for Title II-A adults and adult welfare 
recipients and for dislocated workers under EDWAA address the DOL's need for 
accurate, consistent data that will permit confidence in the measures and comparisons of 
program performance across States and SDAs within States. Once the follow-up data 
have been collected, they must be reported to the DOL on the JTPA Annual Status 
Report (JASR) for Titles II-A terminees, and the Worker Adjustment Program Annual 
Program Report (WAPR) for EDWAA terminees. Separate WAPRs must be submitted 
for (1) each SSA; (2) participants and terminees in statewide, regional, or industrywide 
projects funded under Section 302(c)(1)(B) of EDWAA; and (3) participants and 
terminees in projects funded under Secretary's National Reserve Grants (Section 302(a) 

(2). 

The following data must be reported in Section III of each report: 

• Employment Rate At Follow-up 

• Average Weekly Earnings of Employed At Follow-up (JASR) 
Average Hourly Wage of Employed at Follow-up (WAPR) 

t Average Number of Weeks Worked in Follow-up Period 

• Sample Size 

• Response Rate 

Data for the first three items (employment rate, earnings, weeks worked) must be 
adjusted for nonresponse bias, as described in the previous section of the TAG. The 
"Report" option on the software to support JTPA follow-up can be used to produce a file 
with the required information, in a format that can be printed for the required 
JASR/WAPR. (See Appendix H.) 

Exhibit 3,1 presents Section III. Follow-up Information from the cunent JASR. 
Complete copies of the PY 1989 JASR and WAPR, along with instructions for 
completing them, appear as Appendices A and B respectively. 
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Exhibit 3.1 



JASR Section III. Follow-up linformation 
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You must provide the DOL with completed follow-up information for the 
program year no later than 45 days after the end of the program year-that is, by August 
15, Because of the scheduled time period allowed for reporting, the data reported to the 
DOL by August 15 may not be completely edited. Therefore you should report updated 
data to the DOL as soon as possible after new data are entered and edited into the MIS. 
The DOL will include the updated information in the database used for devising the 
regression model worksheets and in its own annual reports. 



Reporting Follow-up Results for State and Local Management 

As discussed in Section 1.3, States and SDAs can use the follow-up data as a 
valuable management tool. In addition to reporting the required JASR/WAPR 
information regarding employment status and earnings 13 weeks after termination, 
reporting these results in relation to participant characteristics, programs, and service 
providers is very important. Where follow-up is centralized, it is especially important 
that States provide comprehensive follow-up data to the SDAs within the State in a 
timely manner. 
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Reporting formats. State and local JTPA administrators can determine how best 
to combine, or aggregate, these data to present follow-up results that can be used for 
making meaningful comparisons and program decisions. They might, for example, 
combine results across SDAs, across years, or across similar programs. Clearly labeled 
tables or graphs, and prose summaries should present relevant data in a concise way. 
Exhibit 3.2 provides examples of reporting formats that can be meaningful for follow-up 
data from States, individual SDAs (if the numbers in each group are sufficient), or 
combinations of SDAs in a State. 
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Exhibit 3.2 



Sample Report Formats for JTPA Follow-up Data 
Suggested Reports to SDAs 



1. Summaiy of SDA Data Compared to State Totals 

Is Difference 



SDA State Average Significant?* 

Sample Size 1000 600 yes 

Number of Respondents 900 450 yes 

Response Rate 90% 75% yes 

Follow-up E:nployment Rate 60% 61% no 

Average Weekly Earnings $322 $307 no 

Average Weeks Worked 10.3 6.8 yes 

2. Comparison of Follow-up Data by SDA Contractors 

Is Difference 

Contractor A SDA Average Significant?' 



Sample Size 

Number of Respondents 
Response Rate 
Follow-up Employment Rate 
Average Weekly Earnings 
Average Weeks Worked 



3. Comparison of FoIIow-up Data by Type of Program within SDA 

Is Difference 

Program A SDA Average Significant?* 

Sample Size 

Number of Respondents 
Response Rate 
Follow-up Employment Rate 
Average Weekly Earnings 
Average Weeks Worked 



* refers to statistical significance 

« 
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Exhibit 3.2 (Continued) 



OTPA POSTPROGRAH FOLLOW-UP REPORT 
PY '87.- FOURTH QUARTER 

Table 1 orovldes statewide summary data on the postprogram •mplbyment and earnings 
exoerlences for several Title IIA subgroups. As the table Indicates, 4,157 Tltlt IIA 
adult terSlnees were Interviewed during this period. Sixty-eight percent (68X) werji 
employed during the thirteenth week following termination and were earning ;n avtrage 
weekly wage of $195.33. Termlnees reported an average of 9 weeks employment during tne 
13 week period following 3TPA participation and, for those reporting any employment, 
worked an average of 34 hours per week. 

In examining subgroups of respondents, women were as likely as men to be •inP]py;«*» 
were earning an average of $60 less per week than men. Dropouts were less likely to be 
employed thirteen weeks after JTPA, and earned about $20 less per week than non-dropouts 
when employed. Older workers (age 55 and over) experienced t!;^ioy"**veraae weekly 
earnings ($148.30). The employment and earnings experience of AFDC/WEOP recipients Is 
quite similar to the overall population of respondents. 

TA8LE 1 

SELECTED OUTCOMES 8Y CHARACTERISTICS OF JTPA TITLE IIA FOLLOW-UP RESPONDENTS 



Number of 



Percent 
Employed 



Average 
Weekly 



Average 
Weeks 



Average 
Weekly Hours 



Women 
Hen 


2.507 
1,650 


681 
68 


»r ILLUMJ 

$169.98 
229.52 


9.0 
9.2 


32.6 
35.4 


Dropouts 
Non-dropouts 


812 
3,345 


59 
70 


178.20 
198.93 


7.7 
9.4 


32.7 
34.0 


Minorities 
Whites 


619 
3,538 


51 
72 


186.14 
196.98 


7.0 
9.6 


33.5 
33.8 


Handicapped 
Non-Handicapped 


723 
3,434 


66 
68 


185.53 
197.25 


8.9 
9.1 


31.9 
34.1 


Age 55 and Over 
Under Age 55 


526 
3,631 


62 
69 


148.30 
201.79 


8.5 
9.2 


28.4 
34.5 


AFDC/WEOP Recipient 
Non-AFDC/WEOP 


1,060 
3,097 


65 
68 


189.11 
196.90 


8.9 
9.1 


33.9 
33.7 


Welfare Recipient 
Non-Welfare 


2,338 
1,819 


63 
73 


198.09 
193.06 


8.4 
9.7 


34.0 
33.5 


Total Respondents 


4,157 


68 


195.33 


9.1 


33.8 
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Exhibit 3.2 (Continued) 



Job Training Council 
13-Week Earnings and Employment Status by M^or Activity 



# of % Employed Avg. Wkly. Earnings 

Responses at 13WK of Those Employed 

Activity 

Classroom Training 159 67% $248.68 

Customized Traim'ng 73 82% $282.38 

OJT 89 81% $267.92 

Supplementary Work 28 46% $188.08 

Tuition Assistance 110 81% $316.11 

Job Search 51 69% $253.82 

TOTAL ALL RESPONDENTS 510 74% $272.02 
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JobMille Follou-Up Results 
Progran Year Ending June 3Bi 1987 

Figure I: Enploywent Status by Age, Gender A Education; 

13 Ueeks After TerMination 
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Total (N=55e) 
Post H.S. (N=96) 
H.S. Grad. (N=358) 
Dropout (N=96) 
Fenale (N-325) 
Male (N=225) 
Youth (N:=95) 
Uelfare (N=148) 
Adult (N=3e7) 




5 
I' 



id 20 38 



Prepared by EnployMent 
and Training Institute 



40 50 60 

Percent 
Employed 



100 
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Jobville Follow-Up Results 
PrograM Year Ending June 38 « 1987 

Figure If: Average Ueeklij Gross Uage of Enployed by Age« 
Gender d Education; 13 Weeks After Temination 



388 T 



258 



288 



Dollars 158" 



188 



58 



8 




Adult Uelfare Youth Hale Fenale Dropout 
(N^7)(N=148) CN=95) (H=225) (H=325) (H=96) 



H.S. 
Grad. 
CN=358) 



Post Total 
H.S. (N=558) 
(N=96) 
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Limitations In the data. Despite the value of using follow-up information, even 
completely accurate information might not be meaningful. The significance of the data 
depends on the size of each group being examined. A small SDA that wants to evaluate 
different programs in terms of their follow-up employment rates might not be able to 
make reliable conclusions based on small numbers of terminees from each program, 
even if all terminees were followed up. 

For example, suppose that in one SDA 60 terminees participated in classroom 
training programs, and 60 participated in OJT programs. If 50% (30) of those in 
classroom training were working for pay in week 13, then it would take 65% (39) of the 
60 in OJT for OJT to be judged significantly more effective than classroom training. A 
smaller difference could be attributed to random chance: it might have just happened 
that individual employers for classroom training terminees had business slowdowns. 

These limitations also apply to follow-up data that are not required by DOL--for 
example, responses to attitudinal questions. It is extremely important not to 
overinterpret the intense response of one or a small number of respondents. Program 
decisions must not be based on one terminee's opinion of a classroom training program 
(favorable or unfavorable); however, if all or a substantial number/percentage of 
terminees concur, the results might appropriately affect decisionmaking. 

In interpreting the follow-up data, especially from small samples, it is important to 
ask whether the participants in one program were different from those in another, in a 
way that might account for differences in follow-up results. For example, contractors 
serving harder-to-serve participants can be expected to have somewhat lower follow-up 
employment rates. They should not be penalized for reasonable deviations from average 
follow-up rates. 

Exhibit 3.3 presents the minimum numbers of follow-up terminees required in a 
group for percentage differences in follow-up employment rates between groups to be 
meaningful. Note that the numbers in the right column of the table refer to respondents 
(completed Interviews) only, not nonrespondents in the sample. 
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Exhibit 3.3 



Minimum Sample Size for Differences In Follow-up Employment Rates 
Between Two Groups to be Meaningftii* 

Minimum Number of Respondents in Smallest 
Difference in % Employed Group for Difference to be Meaningful 



5% 


540 


6 


375 


8 


211 


10 


135 


12 


94 


15 


60 


20 


34 


25 


22 


30 


15 


40 


10 



'These numbers and percentages apply (o differences in employment rates only. Corresponding numbers for weekly earnings and 
weeks worked cannot be presented because only aggregate SDA data are available to DOL However, they can be calculated at the 
State and local levels, where individual SDA data are available. 



For example, if two training programs had follow-up employment rates of 75% 
and 55% respectively, there would be a 20% difference. This difference would be 
significant, and therefore meaningful and useful for program decisions, only if there were 
at least 34 follow-up respondents in each group. (Note in Exhibit 3.3 that a difference of 
20%, in the left hand of the table, requires at least 34 respondents in each g:oup.) For 
small SDAs, it may take a substantial time to compile the data needed for making 
reliable decisions, where differences are small: and programs may change during that 
period. On the other hand, States may be ab. .. to support local decisionmaking by 
combining follow-up data from similar training programs across SDAs in the State. 
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Section 4 - Assuring the Accuracy and Validity of FoUow-up Data 

(VaUdation) 



This section presents validation in the context of JTPA follow-up data collection. 
It includes (1) an overview which discusses the importance of validation, the need for 
communicating validation activities and results to all involved in the follow-up process, 
and the relationship between validation and centralization or decentralization of follow- 
up; (2) a description of various validation methods; and (3) other issues related to 
validation. 



4.1 Overview of Validation 

Validation is essential in a performance management system like JTPA that 
rewards programs based on outcomes. To award incentives fairly and to make intelligent 
management decisions, it is critical that the follow-up data be accurate and that 
procedures for collecting the data be consistent and meet the Federal requirements. 
Although it is equally important for termination data to be accurate and valid (since they 
are also used for management decisions and to determine incentives), MIS records are 
typically available for documenting and monitoring termination outcomes. Follow-up 
data, however, depend on a respondent's self-report of employment status and earnings. 
Validation is especially needed to ensure consistency across SDAs in decentralized 
States, because locating and interviewing respondents can be done in different settings 
with different procedures and outcomes. Even in centralized States, there is a need to 
ensure consistency in the contact information provided by SDAs. 

This TAG has presented data collection procedures designed to maximize 
response rates and enhance the consistency and accuracy of the follow-up information 
gathered. While these procedures go a long way toward ensuring accurate data, 
additional steps can and should be taken to validate the follow-up data, from the 
beginning to the end of the process. The need for validation and its cost depends upon 
several factors, including the organization of the follow-up (whether the State or SDA 
collects the data), the extent cf the follow-up effort, and the size of the respondent 
population. Although DOL does not require validation of follow-up data. States should 
assess the need for validation in terms of these factors, and implement consistent 
procedures across SDAs that produce the most accurate follow-up data possible. 
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Deflning validation. For follow-up data collection, validation includes any and 
all activities undertaken to assess and assure the accuracy of the data collected, 
reported, and ultimately used for important management decisions* Broadly defined, 
validation includes all monitoring and quality control procedures designed to assure that 
appropriate and consistent follow-up methods are used. More narrowly defined, 
validation includes activities designed to verify, against additional sources of 
information, the accuracy and corisistency of the follow-up data itself. Both types of 
validation activities are relevant to insuring high quality follow-up data. 

Accuracy of the follow-up data refers to the correctness of the figures reported. 
Do the reported figures correctly capture each of the follow-up data items, including 
employment status, weekly earnings, and weeks worked? If a respondent is reported as 
"employed" at 13 weeks but was actually unemployed, then the data are inaccurate. 
Although data reported to DOL may contain some minimal error, it is important that the 
extent of error is small and consistent-that it does not vary among SDAs. 

Implications of centralization on need for validation. The need for validation, 
the extent of the validation effort required, and its expense will likely be greater if the 
responsibility for follow-up is decentralized. When each SDA performs its own 
follow-up, validation becomes particularly important because States must guard against 
inconsistent data collection and quality control procedures, as well as inaccuracy from 
biased reporting. States and SDAs within each State must be assured that all of the 
other SDAS have implemented the follow-up data collection procedures exactly and that 
the reported data accurately represent the employment experiences of terminees during 
the follow-up period. Validation should be sufficiently extensive to let each actor in the 
system rest easy. Decentralized States will need to implement a more extensive, 
centralized validation to guard against the inconsistencies and biases inherent in follow- 
ups conducted at the SDA level. More extensive validation will be more expensive as 
well. Adequate training, setup, supervision, and monitoring of follow-up procedures and 
gathering data from additional sources of information from SDAs will increase costs. 

On the other hand, when follow-up is centralized at the State level-regardless of 
whether interviewing is in-house or contracted out-validation activities can be more 
limited, and therefore less expensive. The State's liability in granting awards based upon 
SDAs' follow-up performance encourages reporting as accurately and consistently as 
possible to preclude complaints from SDAs based upon inequitable treatment. The 
primary validation required may be procedural such as supervising staff or monitoring 
outside contracts. Activities would include supervising data collection f\ctivities and 
monitoring data entry, records, and reports. Because a centralized C uses consistent 
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42 VaUdation Methods 



This section describes a variety of validation methods, which fall into two 
principal categories: (1) monitoring and supervising activities and (2) validating follow- 
up data against additional sources of information* 

Monitoring and Supervising Follow-up Data Collection 

Monitoring follow-up data collection is critical to ensuring that appropriate 
contacts are being made, that all necessary procedures are being followed correctly and 
consistently, and that adequate training and technical assistant*, are prcdded. 
Monitoring activities ensure that all steps in the follow-up process are conducted 
correctly, from beginnii. ^ to end: sampling, timing, locating and contacting appropriate 
participants, interviewing, coding and entering data, preparing reports, and transmitting 
information in a timely fashion among States, SDAs, contractors, and the DOL. The 
following procedures should be standard practice, whether the follow-up is centralized, 
decentralized, conducted in-house or contracted out. 

Specify a number or percent of interviews or records to monitor periodically. The 
percent will depend on the size of the follow-up effort. In a centralized follow-up or in a 
large SDA, a sample of at least 5-10% of the follow-up interviews for each interviewer 
should routinely be monitored. In a small SDA, at least 10 interviews for each 
interviewer should be monitored to ensure that they are being conducted properly. If 
the follow-up is contracted out, procedures such as these should be specified in the RFP 
and contract. 

• Listen in on interviews to make sure all questions are asked 
appropriately. Record the responses, and compare them to the 
responses recorded by the interviewer. Check to make sure 
ambiguities and inconsistencies were clarified in a neutral fashion. 
Provide interviewers with immediate feedback. Monitoring 
interviews in this way is an opportunity not only to validate, but to 
praise, instruct, and learn about effective interviewing techniques. 

• Review contact logs and completed interview records. Review the 
logs to ensure that adequate and timely attempts are made to reach 
terminees. Examine interview records for completeness and 
accuracy. 
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• Supervise and verify data entry. Double data entry (100% key 
verification) is recommended to ensure accuracy. 

• Check data against the DOL limits described in Section 2.6, 

• Establish additional MIS checks for internal consistency and reasonableness, 
based on local labor market conditions and common sense. Identify and 
resolve inconsistent or out-of-range responses-for example, set limits for 
hourly wages at $3.80 • $15.00, and for hours worked during the 13th week at 
0 -70. Compare paper records with MIS records to identify and resolve 
discrepancies-for example, large disparities between placement and follow-up 
data. 

• Verify the summaiy follow-up data that will be reported to the DOL against 
individual terminee records. This may involve comparison of paper records 
with MIS records. 

• Check the MIS software to ensure that it is accurately recording data and 
making appropriate edits. 

• Conduct periodic monitoring visits to ensure compliance if follow-up is 
done by SDAs or outside contractors. Follow-up should be an integral 
component of each State's monitoring activities. Monitoring should 
identify problems in procedures and data collected, and recommend 
corrective actions to ensure accuracy of the follow-up results. 

• If the above validation activities reveal a substantial number of discrepancies 
( > S% of the data examined), expand your validation activities to include a 
larger percentage of the follow-up sample. Monitoring or review of 100% of 
the interviews or records may, in fact, be warranted if discrepancies are very 
large. 

Validating Follow-up Data Against Additional Sources of Information 

Comparing follow-up data with information obtained from other sources is a 
useful way of validating the follow-up effort, particularly for decentralized States who are 
granting incentives based on the follow-up results. Several methods are appropriate. As 
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with monitoring activities, identify a specific percentage or number of the foUow-up 
sample to validate. 

• Re-interview respondents to assure they have been previously 
contacted and to verify their responses. Be sure to alert terminees 
during the initial interview that they might be called again by 
another person. 

• Contact employers, by phone or mail, to verify the follow-up 
employment and wage data provided by participants. If you contact 
employers, be sure to obtain signed permission from terminees to do 
so, (see Exhibit 2.4, Employer Release Form) and to establish 
cooperative relationships with local employers. When contacting 
employers, it is very important not to release confidential 
information a Jut participants or to disrupt or irritate employers 
who could be the source of future JTPA placements. It is also very 
important that information provided by employers NOT be used for 
JASR/WAPR reporting purposes in lieu of information provided by 
JTPA terminees. A sample employer verification form appears as 
Exhibit 4.1. 

• Ask additional questions and compare responses to MIS records to 
verify that correct participants have been interviewed and that 
responses are accurate. For example, have interviewers ask 
terminees for their birthdates, but do not provide this information 
to the interviewers. Then match the data to the MIS records. 

• In centralized States where SDAs do any independent follow-up, 
compare SDA data to State follow-up data. If two separate follow- 
ups are done by the State and SDA, participants should be informed 
that they may be called more than once and results can be 
compared. 

• Compare follow-up data to Unemployment Insurance (UI) wage 
records. Employers in all States are required to report the total 
wages paid to employees during each calendar quarter. This 
information, called the employee's UI Wage Record, can be 
compared to the wage information provided by participants as an 
approximate validation measure. The advantages of using UI 
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records are that it uses information that is already available, it is 
inexpensive, and it is relatively easy to do. The disadvantages are 
that the wage records do not include the specific JASR information 
on the 13th follow-up week, are not available until the second 
quarter after the JTPA follow-up period ends, and may not include 
all JTPA terminees (self-employed, out-of-State, and "underground" 
jobs may not be reported) or all income and employment for 
terminees (tips, second jobs). 
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Exhibit 4.1 



Sample Employment Verificatiou Form 

Company name 

Address . 

City, State, Zip 

Emp loyee^ 

Social Security Number 

Start date ^Currently working? Yes No 

Total number of weeks worked since start date 

Hourly wage since start date , 

Was Employee working the week of 

Number of Hours the Employee worked the week of__ ^Hrs. 

Employees Hourly Wage the week of $ 

Employees earnings before deductions from Tips $ ^Overtime $ 

Bonuses $ Commissions $ the week of 

Job Title 

Brief Job Description 



Please return to Camden County Employment & Training Center 
Thank you. 

*This form is used by the Camden County Employment and Training Center, Cainden, NJ. 
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43 Other Validation Issues 

Other validation issues that States and SDAs need to address include identifying 
and resolving discrepancies found when validating follow-up data, as well as the 
implications of invalid data. 

Identifying and Resolving Discrepancies 

Validating follow-up data with information obtained from other sources is a 
comparative process. Information provided by a respondent is compared with another 
supposedly accurate measure. If there are discrepancies, data should be changed to 
provide the most accurate information, if this is possible to determine. 

Unfortunately, the data used to validate a terminee's responses may be no better 
than the data provided by the terminee. Suppose the terminee reports earning $150.00 
during the 13th follow-up week. The employer reports that the terminee was paid 
$120.00 for that week. Who is correct? 

It is possible that both are correct. The terminee may have been paid only 
$120.00 by the contacted employer, but might have earned $30.00 in tips or at a second 
job. It is also possible that both are wrong. For these reasons, dcta should not be 
changed without recontacting the respondent, providing information about the 
discrepancy, and asldng for assistance in its resolution. 

s 

Comparisons with Unemployment Insurance (UI) records can also lead to the 
identification of apparent discrepancies. UI records may not match. It must be 
remembered that not all employers contribute to the UI pool, that individuals often earn 
tips or other income that are not reliably reported to UI, and that UI records are 
sometimes incorrect. 

Another type of validation involves reint^'.rviewing the respondent. Because 
human memory is imperfect, if you were asked, "How much money did you earn last 
week?" and a month later were asked the same question, you would most likely give two 
different numbers. (You might also believe that your original answer was more accurate 
since it was closer to the week in question.) 

Validation efforts will frequently lead to the identification of discrepancies. If the 
discrepancies are in only a small percentage of cases (i.e., fewer than 10%) and are 
small and not always in the same direction (i.e., systematic over-reporting of income), 
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they can be tolerated because they will not seriously distort aggregate statistics. 
However, unresolved discrepancies can seriously reduce the value of follow-up effort if 
any one of these conditions occurs: 

• more than a small percentage of cases have discrepancies 

• any of the discrepancies are large 

• the discrepancies are always in the same direction 

If respondents consistently under- or over-report their earnings, data will be 
biased and conclusions drawn from these data may not be meaningful. Validations that 
result in the detection of a consistent directional bias demand further action. The first 
thing to do is to ascertain that the source of the validating information is unbiased. As 
previously mentioned, UI records and employers may not include tips or other income. 
Recontacting respondents is essential to get more accurate employment and earnings 
information and to determine the source of any inaccu/acies. Once you know why 
incorrect data were provided, you can develop new procedures or modify old procedures 
to avoid recurrence of such problems. 

Large discrepancies may indicate misinterpretations or improper administration of 
the questions. Once again, the source of the validating information should be checked to 
make certain that critical components are not being excluded or that improper items are 
not being included. Recontacting respondent is again necessary to determine the source 
of the inaccuracies and to permit modification of old procedures or the development of 
new procedures to avoid recurrence of these problems. 

Implications of Invalid liata 

Random discrepancies have no more effect on the value of the follow-up data 
than sampling enor. However, systematic discrepancies, or bias, are more serious; they 
can lead to erroneous conclusions about program effectiveness. The detection of bias in 
the data demands that procedures be reviewed and changed to avoid this bias. Such 
information should be shared with the Department of Labor to permit proper 
interpretation of the employment and income data you provide them. 
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In addition, discrepancies identified during validation efforts must be 
communicated with the State, SDA, or contractor staff involved. Corrective action 
should be designed to eliminate the problems found during monitoring and validation. 
Appropriate corrective action will depend upon the unique situation of the State or SDA 
involved, and might relate to various components of the follow-up system, ranging from 
interviewer training, to procedures for locating terminees, to procedures for entering and 
analyzing the data. 

Regardless of the outcome of your validation efforts, the DOL should be 
informed of their results. It is very useful to have an indication of validity to interpret 
data properly. When you are unsure of the level of validity, it is difficult to draw 
meaningful conclusions and make recommendations based on these data. 
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Appendix A 

JTPA Annual Status Report and Instructions (1989 JASR) 

[NOTE: The PY 1990 JASR and Instructions should be inserted here when it becomes available.] 
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Purpose . The JTPA Annual Status Report (JASR) displays 
cumulative data on participation, termination, performance 
measures and the socio-economic characteristics of all 
terminees on an annual basis. The information will be used 
to determine levels of program service and performance 
measures. Selected information will be aggregated to provide 
quantitative program accomplishments on a local. State, and 
national basis. 

General Instructions . The Governor will submit: 1) a com- 
bined statewide JASR for Title III Formula and Discretionary 
National Reserve (Column D only); 2) for Title ii-A (Columns 
A-C) a separate JASR for each designated Service Delivery 
Area (SDA) and 3) a separate Statewide JASR summary of the 
SDA report data. (This Statewide JASR summary of individual 
SDA data should not be submitted by single-SDA States.) 
Grantees may determine whether the reports are submitted on 
JASR forms or as a computer printout, with data, including 
signature and title, date signed and telephone number, 
arrayed as indicated on the JASR form. if revisions are made 
to the JASR data after the reporting deadline, revised copies 
of the JASR should be submitted to DOL as soon as possible 
according to the required reporting procedures. Submittal of 
one or more JASRs with revised information for Total Partici- 
pants, Total Terminations and/or Total Program Costs (Federal 
Funds) usually will require submittal of a revised JSSR which 
includes the final quarter of the same program year. 

NOTE: For JASR reporting purposes. Title ll-A shall refer to 
programs operated with funds authorized under Section 202(a) 
of the Act or otherwise distributed by the Governor under 
Section 202(b)(3) (6%) of the Act— incent ive grants for 
service to the hard-to-serve and programs exceeding 
performance standards. (Concentrated Employment Programs 
(CEPs) should report total Title ii-A program expenditures of 
78% funds, special supplemental allocations, and 6% incentive 
grants.) Do not include data on (6%) funds authorized under 
Section 202(b)(3) for technical assistance. Participants and 
expenditures under Title I, Sections 123 (8%) and 124 (3%), 
and expenditures under Title II, Section 202 (b)(4) (5%) and 
any participants, if applicable, are likewise excluded from 
the JASR. 

NOTE: Participant and expenditure information under Title 
II-B, Summer Youth Employment and Training Program (SYETP) , 
is also excluded from the JASR. 

SDAs should not terminate from Title ii-A youths who 
participate in the Title II-B Summer Program unless they are 
not expected to return to Title ll-A for further employment, 
training and/or services. 

If these youths receive concurrent employment, training 
and/or services under both Titles ll-A and II-B, they 
are to be considered participants in both titles for 
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purposes of recording actual number of weeks 
participated, dollars expended, and other pertinent 
data . 

If, however, these youths do not receive Title li-A 
employment, training and/or services while 
participating in Title ll-B, this period is not to be 
included in the calculation of actual number of weeks 
participated in Title Il-A at Line 29, Column C. 

The reporting period begins on the starting date of each jtpA 
program year, as stated in Section 161 of the Act. Reports 
are due in the national and regional offices no later than 45 
days after the end of each program year. Two copies of the 
JASR are to be provided to: 

Employment and Training Administration 
U.S. Department of Labor , 
ATTN: TSVR — Rm . S-5306 
200 Constitution Avenue, N.W. 
Washington D.C. 20210 

At the same time an additional copy of the jasr is to be 
provided to the appropriate Regional Administrator for 
Employment and Training in the DOL regional office that 
includes the state in which the JTPA recipient is located. 

3. Facsimile of Form . See the following page. 

^- Instructions for Com pleting the JTPA Annual stat us Report 
(JASR) . ' 

3. State/SDA Name, Number and Address 

Enter the name and address of the state agency that will 
administer the grant recipient's program (Title m 
report). Enter the name, ETA assigned SPA nmabe r and 

address of the designated SDA subrecipient , as 

appropriate (Title ii-a report). 

b . Report Period 

Enter in "From" the beginning date of the designated JTPA 
program year and enter in "to" the ending date of that 
program year . 

Signature and Title (at bottom of the page) 

The authorized official signs here and enters his/her 
title. 
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d . Date Signed 

Enter the date the report was signed by the authorized 
official. 

e. Telephone Number 

Enter the area code and telephone number of the 
authorized official. 

5. General Information . For purposes of the JASR, the Total 
Adults and Adults (Welfare) columns will include terminees 
age 22 years and older. Thus, the column breakouts are based 
strictly on age rather than on program strategy. The youth 
column will include terminees who were age 14-21 at the time 
of eligibility determination. The Dislocated Workers column 
may include adults and youth, as applicable. 

Unless otherwise indicated, data reported on characteristics 
of terminees should be based on information collected at the 
time of eligibility determination. 

CHARACTERISTICS INFORMATION OBTAINED ON AN INDIVIDUAL AT THE TIKE 
OP ELIGIBILITY DETERMINATION FOR THE RECIPIENT'S JTPA PROGRAM 
SHOULD NOT BE UPDATED WHEN THE INDIVIDUAL TERMINATES FROM THE 
JTPA PROGRAM. 

COLUMN HEADINGS 

Column A Total Adults 

This column will contain an entry for each appropriate item for 
all adult participants in Title II-A only. 

Column B Adults (Welfare) 

This column will contain an entry for each appropriate item for 
adult participants in Title II-A who were listed on the welfare* 
grant and were receiving cash payments under AFDC (SSA Title IV). 
General Assistance (State or local government), or the Refugee 
Assistance Act of 1980 (PL 96-212) at the time of JTPA 
eligibility determination. For reporting and performance 
standards purposes, exclude those individuals who receive only 
SSI (SSA Title XVI) from entries in Column B. 

NOTE: Column B is a sub-breakout of Column A; therefore. Column 
B should be less than or equal to Column A for each line entry. 

Column C Youth 

This column will contain an entry for each appropriate item for 
all participants, aged 14 - 21, in Title II-A only. 
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Column D Dislocated Workers 

This column will contain an entry for each appropriate item for 
all participants in Title III Formula and Discretionary National 
Reserve who were determined to be eligible dislocated workers. 

NOTE: Columns A, B, and C apply to Title II-A only. Column D 
applies to Title III orily. All information regarding a given 
participant must be entered in the same column, e.g., Column C 
for a youth in Title II-A. 

The sum of the entries (all SDAs in a State) in Columns A and C, 
Item I. A., Total Participants, of the JASR should equal the entry 
in Column A, Item III.A.l., SDA Participants, of the JSSR, for 
the same recipient, that includes the final quarter of the same 
program year. The entry in Column D, Item I. A. of the Statewide 
JASR for Title III should be the sum of the entries in Columns B 
and C, Item III. A. of the JSSR, for the same recipient, that 
includes the final quarter of the same program year for the same 
grant . 

The sum of the entries (all SDAs in a State) in Columns A and C, 
Item I.B., Total Terminations, of the JASR should equal the entry 
in Column A, Item III.B.I., SDA Terminations, of the JSSR, for 
the same recipient, that includes the final quarter of the same 
program year. The entry in Column D, Item l.B. of che statewide 
JASR for Title III should be the sum of the entries in Columns B 
and C, Item III.B. of the JSSR, for the same recipient, that 
includes the final quarter of the same program year for the same 
gr?int . 

SECTION I - PARTICIPATION AND TERMINATION SUMMARY 



Section I displays the program's accomplishments in terms of the 
total cumulative number of participants in the program and the 
number and types of terminations from the program, as of the end 
of the reporting period. 

Entries for items I. A. and l.B. are cumulative from the beginning 
of the program year through the end of the reporting period. 

Item I. A. Total Participants 

Enter by column the total number of participants who are or were 
receiving employment, training or services (except post- termina- 
tion services) funded under that program title through the end of 
the reporting period, including both those on board at the begin- 
ning of the designated program year and those who have entered 
during the program year. if individuals receive concurrent 
employment, training and/or services under more than one title, 
they are to be considered participants in both titles for 
purposes of recording actual number of weeks participated, 
dollars expended, and other pertinent data. 
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"Participant" means any individual who has: (1) been determined 
eligible for participation upon intake; and (2) started receiving 
employment, training, or services (except post- termination 
services) funded under the Act, following intake. Individuals 
who receive only outreach and/or intake and initial assessment 
services or postprogram follow-up are excluded. 

participants who have transferred from one title to another, or 
between programs of the same title, should be recorded as 
terminations from the title or program of initial participation 
and included as participants in the title or program into which 
they have transferred, unless they are to be considered 
concurrent participants in both titles or programs. 

Item I.B. Total Terminations 



Enter by column the total number of participants terminated after 
receiving employment, training, or services (except post- termina- 
tion services) funded under that program title, for any reason, 
from the beginning of the program year throuc-h the end of the 
reporting period. This item is the sum of Ite.-ns T.B.I, through 
I • B • 3 • 



Termination" means the separation of a participant from a given 
title of the Act who is no longer receiving employment, training, 
or services (except post- termination services) funded under that 
title. NOTE: Individuals may continue to be considered as par- 
ticipants for a single period of 90 days after last receipt of 
employment and/or training funded under a given title. During 
the 90-day period, individuals may or may not have received 
services. For purposes of calculating average weeks 
participated, this period between "last receipt of employment 
and/or training funded under a given title" and actual date of 
termination is defined as "inactive status" and is not to be 
included in Line 29. 

Individuals who initially participate in Title III Formula funded 
activity and subsequently participate in Title ill Discretionary 
National Reserve funded activity (and conversely) , FOR THE 
COMPLETION OF THE INITIALLY DETERMINED TRAINING OBJECTIVE, may be 
considered to be concurrent participants in both Title III 
programs. The type of termination determined for the subsequent 
program also should be recorded for the initial program for such 
participants. 

For purposes of calculating average weeks participated for such 
concurrent Title III program participants, the period between 

last receipt of employment and/or training funded under a given 
Title III program" (i.e.. Formula or Discretionary National 
Reserve) and actual date of termination from that Title ill 
program is defined as "inactive status" and is not to be included 
in Line 29. 



110 



Item I.B.I. Entered Unsubsidized Employment 



Enter by column the total number of participants who, at termina- 
tion, entered full- or part-time unsubsidized employment through 
the end of the reporting period. Unsubsidized employment means 
employment not financed from funds provided under the Act and 
includes, for JTPA reporting purposes, entry into the Armed 
Forces, entry into employment in a registered apprenticeship 
program, and terminees who became self-employed. 

Item I.B.I. a. Also Attained Any Youth Employability Enhancement 

Enter the total number of youth who (1) entered unsubsidized 
employment. Item I.B.I. , and (2) also attained any one of the 
five youth employability enhancements (as enumerated in the 
instructions for Item I.B.2. below and defined in Appendix C) . 
This item is a sub-breakout of item I.B.I. 

Item I.B.2. Youth Employability Enhancement Terminations 

Enter the total number of youth who were terminated under one of 
the Youth Employability Enhancements through the end of the 
report period. "Youth Employability Enhancement" means an out- 
come for youth, other than entered unsubsidized employment, which 
is recogn.^iid as enhancing long-term employability and contribu- 
ting to tiie potential for a long-term increase in earnings and 
employment. Outcomes which meet this requirement shall be 
restricted to the following: (1) Attained PIC-Recognized Youth 
Employment Conpetencies (two or more); (2) Entered Non-Title II 
Training; (3) Returned to Pull-Time School; (4) Completed Major 
Level of Education; or (5) Completed Program Objectives (14-15 
year olds) . 

NOTE: For reporting purposes, a youth shall not be counted in 
Item I. B.2. if s/he entered unsubsidized employent , and shall be 
counted in only one of the five categories enumerated above, even 
though more than one outcome may have been achieved. 

Item I.B.2.a. Attained PIC-Recognized Youth Employment 

Competencies 

Enter the total number of youth who, at termination, have demon- 
strated proficiency as defined by the PIC in two or more of the 
following three skill areas in which the terminee was deficient 
at enrollment: pre-employment/wor k maturity, basic education, or 
job-specific skills. Competency gains must be achieved through 
program participation and be tracked through sufficiently 
developed systems that must include: quantifiable learning 
objectives, related curricula/training modules, pre- and post- 
assessment, employability planning, documentation, and certifi- 
cation. This item is a sub-breaKout of item I.B.2. 
During py '88, competency attainment in one skill area will meet 
this definition. The entry for Item I.B.2.a. must be equal to or 
smaller than the entry for Line 38 in PY 1988 and PY 1989. 

Rev. 9/89 
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NOTE: Terminees who have attained a competency in basic 
education skills and/or job specific skills through training 
funded under 8% prograirs and/or cooperative agreements may be 

counted in Item I.B.2.a. provided such training was for 
completion of a training objective initially determined while in 
a youth employment competency sys^-em operated under 78% funds. 

Appendix B defines the minimal structural and procedural elements 
of a sufficiently developed youth employment competency system as 
well as the minimal requirements for ensuring consistency in the 
reporting of competency attainment in the pre~employment/work 
maturity skill area. 

Item I.B.2.b. Completed Program Objectives (14-15 year olds) 

Enter the total number of youth (ages 14 and 15 only) who, at 
termination, had completed a program objective. This item is a 
sub-breakout of Item I.B.2. 

NOTE: For Column C, (1) Item I.B.2,b. cannot be greater than 
Item II. 3., (2) the sum of Items I.B.2.a. and I.B.2.b. cannot be 
greater than Item l.B.2. and (3) Item l.B.l. plus item I.B.2. 
plus Item I.B.3. must equal Item I.B. 

Item I.B. 3. All Other Terminations 

Enter by column the total number of participants who were termi- 
nated for reasons other than those in Items l.B.l. and I.B. 2., 
successful or otherwise, through the end of the reporting period. 
Include intertitle transfers here , see notes at item i.e. 

SECTION II - TERMINEE PERFORMANCE MEASURES INFORMATION 

Section ll displays performance measures/parameters information. 
As indicated previously, data reported on characteristics of 
terminees should be based on information collected at time of 
eligibility determination unless otherwise indicated. 

Governors may develop any participant record which meets the 
requirements of Section 629.35(c) and (d) of the JTPA 
regulations. The DOL/ETA Technical Assistance Guid e! The JTPA 
Participa nt Record , dated May 1983, may be used as a refere nce. 

Line item Definitions and Instructions 
Sex 

Line 1 Male 
Line 2 Female 

Distribute the terminees by column according to Sex. The sum of 
Lines 1 and 2 in each column should equal item I.B. in that 
column . 
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Age 






Line 


3 


14-15 


Line 


4 


16-17 


Line 


5 


18-21 


Line 


6 


22-29 


Line 


7 


30-54 


Line 


8 


55 and over 



Distribute the terminees by column according to Age. The sum of 
Lines 3 through 8 in each column should equal Item l.B. in that 
column . 

Education Status 

Line 9 School Dropout 
Line 10 Student 

Line 11 High School Graduate or Equivalent (No Post-High 
School) 

Line 12 Post-High School Attendee 

Distribute the terminees by column according to Education 
Status. The sum of Lines 9 through 12 in each column should 
equal Item l.B. in that column. 

Family status 

Line 13 Single Head of Household with Dependent(s) Under Age 
18* 

Enter the total number of terminees by column for whom the above 
Family Status rlassif icat ion applies. 

Race/Ethnic Group 

Line 14 White (Not Hispanic) 
Line 15 Black (Not Hispanic) 
Line 16 Hispanic 

Line 17 American Indian or Alaskan Native 
Line 18 Asian or Pacific Islander 

Distribute the terminees by column according to the Race/Ethnic 
Groups listed above. For purposes of this report, Hawaiian 
Natives are to be recorded as "Asian or Pacific islander". The 
sum of Lines 14 through 18 in each column should equal Item I.E. 
in that column . 
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Other Barri^irs to Employment 

Line 19 Limited English Language Proficiency 
Line 20 Handicapped 
Line 21 Offender 

Line 22 Reading skills Below 7th Grade Level 
Line 23 Long-Term AFDC Recipient 

Enter the total number of terminees by column for whom each of 
the above Other Barriers to Employment apply, 

U. C. status 

Line 24 Unemployment Compensation Claimant 

Enter the total number of terminees by column for whom the above 
Unemployment Compensation Status classification applies. 

Labor Force Status 

Line 25 Unemployed: 15 or More Weeks of Prior 26 Weeks 
Line 26 Not in Labor Force 

Enter the total number of terminees by column for whom each of 
the above Labor Force status classifications apply. 

Welfare Grant Information 

Line 27 Welfare Grant Type: AFDC 
Line 28 GA/RCA 

Distribute by column the total number of adult and youth welfare 
terminees who, at eligibility determination, were listed on the 
welfare grant and were receiving ca&h payments under AFDC (SSA 
Title IV), GA, General Assistance (State or local government) or 
RCA (Refugee Cash Assistance) under the Refugee Assistance Act 
of 1980 (PL 96-212). If a welfare recipient terminee received 
AFDC cash payments, include such terminee on Line 27. A welfare 
recipient terminee who received cash payments under GA and/or 
RCA, but nojt AFDC, should be included on Line 28. ".he sum of 
Lines 27 and 28 in Column B, Adults (Welfare), should equal item 
I.B. in that column. The sum of Lines 27 and 28 in Column C, 
Youth, should be the same as or less than item l.B. in that 
column . 

Other Program Information 

Line 29 Average Weeks Participated 

Enter by column the average number of weeks of participation in 
the program for all terminees. Weeks of participation include 
the period from the date an individual becomes a participant in a 
given title through the date of a participant's last receipt of 
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employment and/oi training funded under that title. Exclude the 
single period of up to 90 days during which an individual nay 
remain in an inactive status prior to termination. Tine in 
inactive sta .us for all terninees should not be counted toward 
the actual number of weeks participated, inactive status is 
defined as that period between "last receipt of employment and/or 
training funded under a given title" and actual date of 
termination. See note at Item I.B. 

To calculate this entry: Count the number of days participated 
for each terminee, including weekends, from the start date of 
his/her participation in the title until his/her last receipt of 
employment and/or training under that title. For those who 
receive services only, use date of last receipt of such 
services. Divide this result by 7. This will give the number of 
weeks participated for that terminee. sum all the terminees' 
weeks of participation and divide the result by the number of 
terminees, as entered (by column) in Item I.B. This entry 
should be reported to the nearest whole week. 

Line 30 Average Hourly Wage at Termination 

Enter by column the average hourly wage at termination for the 
total number of terminees in Item l.B.l. 

To calculate this entry: Sum the hourly wage at termination for 
all the terminees shown in Item l.B.l. Divide the result by the 
number of terminees shown in Item l.B.l. 

Hourly wage includes any bonuses, tips, gratuities and 
commissions earned. 

Line 31 Total Program Costs (Federal Funds) 

Enter the total accrued expenditures, through the end of the 
reporting period, of the funds allocated to SDAs under Section 
202 (a) of the Act or otherwise distributed by the Governor to 
SDAs under Section 202(b)(3) — incentive grants for services to 
the hard-to-serve and programs exceeding performance standards — 
for Title ii-A programs in Columns A and C (includes costs of 
services to participants aged 14-21), as appropriate, for all 

participants served. Do not include expenditures of funds 

authorized under Section 202(b)(3) for technical assistance. 
Exclude expenditures under Title i, Sections 123 (8%) and 124 
(3%) and Title ll Section 202(b)(4) (5%). Enter the total 
accrued expenditures of Title ill funds received by the Governor 
under Section 301 of the Act in Column D only, for all Title iii 
participants served through the end of the repor ting~per iod . 
Include expenditures of Federal funds only, both Formula and 
Discretionary National Reserve. 

NOTE: Entries will be nade to the nearest dollar. Negative 
entries are not acceptable. The JASR program cost data will be 
compiled on an accrual basis. If the recipient's accounting 
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records are not normally maintained on an accrual basis, the 
accrual information should be developed through an analysis of 
the records on hand or on the basis of best estimates. 

The sum of the entries in Columns A and C, Line 31, Total Program 
Costs, of the JASR (i.e., total for the State's SDAs under Title 
II-A) should equal the entry in Column A, Item I.A.I. , SDA Total 
Program Expenditures, of the JSSR, and the sum of the entries 
(all SDAs in a State) in Column C, Line 31 of the JASR should 
equal the entry in Column A, Item ii. of the JSSR, for the same 
recipient, that includes the final quarter of the same program 
year. The entry in Column D, Line 31, of the Statewide JASR for 
Title III should be th>? sum of the entries in Columns B and C, 
Item I. A. of the JSSR, for the same recipient, that includes the 
final quarter of the same program year for the same grant. 

Line 32 Total Available Federal Funds 

Enter the total Federal funds available for the Title ll-A or 
Title III program described on this report including (1) unex- 
pended funds carried over from previous program years, (2) funds 
allocated or awarded for this program year, and (3) any realloca- 
tion that increased or decreased the amount of funds available 
^ ^ * • ? * . ^ ^ ^ this 

reporting period. Enter 
all Title ii-A funds (Adults and Youth) in Column A and all Title 
III funds in Column D. Title II-A funds include those allocated 
to the SDA by the Governor under Section 202(a) of the Act, as 
well as incentive grants for services to the hard-to-serve and 
for programs exceeding performance standards under Section 
202(b)(3). Exclude funds authorized under Section 202(b)(3) (6%) 
for technical assistance to SDAs and funds received for activi- 
ties under Sections 123 (84) and 124 (3%) and Section 202(b)(4) 
ini /UN '^^^^^ ^""^2 include all Formula amounts under Section 

301(b) and all Discretionary National Reserve awards under 
Section 301 (a) . 

SECTION III - FOLLOW-UP INFORMATION 

Section III displays information based on follow-up data which 
must be collected through participant contact to determine an 
individual's labor force status and earnings, if any, during the 
13th full calendar week after termination and the number of weeks 
s/he was employed during the 13-week period. Follow-up data 
should be collected from participants whose 13th full calendar 
week after termination ends during the program year (the 
follow-up group) . Thus, follow-up will be conducted for 
individuals who terminate during the first three quarters of the 
program year and the last quarter of the previous program year. 

Follow-up data will be collected for the following terminees: 
Title ll-A adults, adult welfare recipients, and Title ill 
dislocated workers (Columns A, B, and D) . No follow-up 
information is required for Title li-A youth (Column C) . 
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The procedures used to collect the follow-up data are at the 
discretion of the Governors. However, in order to ensure 
consistency of data collection and to guarantee the quality of 
the follow-up information, follow-up procedures must satisfy 
certain criteria. (See the Follow-up Guidelines included in 
these JASR instructions, Appendix A.) 

NOTE: Every precaution must be taken to prevent a "response 
bias" which could arise because it may be easier to contact par- 
ticipants who were employed at termination than those who were 
not and because those who entered employment at termination are 
more likely to be employed at follow-up. Special procedures have 
been developed by which SDAs and States can monitor response 
bias. If your response rates for those who were and were not 
employed at termination differ by more than 5 percentage points, 
the follow-up entries for the JASR must be calculated using the 
"Worksheet for Adjusting Follow-up Performance Measures* in the 
Follow-up Technical Assistance Guide. if the response rates 
differ by 5 percentage points or less, the following instructions 
for completing Lines 33-35 may be used. 

Line 33 Employment Rate (At Follow-up) 

Enter by column the employment rate at follow-up. 

Calculate the employment rate by dividing the total number of 
respondents who were employed (full-time or part-time) during the 
13th full calendar week after termination by the total number of 
respondents (i.e., terminees who completed follow-up 
interviews). Then multiply the result by 100. This entry should 
be reported to the nearest one decimal (00.0). 

Line 34 Average Weekly Earnings of Employed (At Follow-up) 

Enter by column the average weekly earnings of those employed 
(full-time or part-time) at follow-up. 

Calculate the (before-tax) average weekly earnings by multiplying 
the hourly wage by the number of reported hours for each 
respondent employed at follow-up; and, if appropriate, add tips, 
overtime, bonuses, etc. Divide the sum of weekly earnings for 
all respondents employed during the 13th full calendar week after 
termination by the number of respondents employed at the time of 
follow-up. Respondents not employed at follow-up rre not 
included in this average. This entry should be reported to the 
nearest whole dollar. *~ 

Weekly earnings include any wages, bonuses, tips, gratuities, 
commissions and overtime pay earned. 

Line 35 Average Number of Weeks Worked in Follow-up Period 

Enter by column the average number of weeks worked. 
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To calculate the average number of weeks worked (full-time or 
part-time) , divide the sum of the number of weeks worked during 
the 13 full calendar weeks after termination for all respondents 
who worked, by the total number of all respondents, whether or 
not they worked any time during this 13-week follow-up period. 
This entry should be reported to the nearest one decimal (00.0). 

Line 36 Sample Size 

Enter by column the size of the actual sample selected to be con- 
tacted for follow-up. (For Title ill only, a statewide sample of 
dislocated workers must be selected. For Title ll-A, i.e., total 
adults and adult welfare recipients, SDA samples must be 
selected . ) 

NOTE: If oversampling was used, the sample size should include 
all those selected, not just the required minimum sample size. 
Those deceased or severely incapacitated to the point of being 
unable to respond at follow-up may be excluded from the sample 
size. 

Line 37 Response Rate 

Enter by column the overall response rate, i.e., the percentage 
of complete surveys obtained. 

To calculate the overall response rate, divide the number of 
terminees with complete follow-up information by the total number 
of terminees included in the follow-up sample (Line 36) and 
multiply by 100. This entry should be reported to the nearest 
whole percent. 

NOTE: Complete follow-up information consists of substantive 
answers to the required follow-up questions and may not include 
don't know", "no answer" or "don't remember". 

SECTION IV - YOUTH EMPLOYMENT COMPETENCY ATTAINMENT INFORMATION 

Section IV displays information relevant to youth employment com- 
petency attainment as defined by the PIC. Regardless of termina- 
tion type, the following data represent the total cumulative 
number of individuals that attained a youth employment competency 
in any of the three skill areas and the numbers of individuals 
who attained a competency in (1) pre-employment/wor k maturity, 
(2) basic education and/or (3) job specific skills. 

NOTE: Terminees who have attained a competency in basic 
education skills and/or job specific skills through training 
funded under 8% programs and/or cooperative agreements may be 

counted in Section IV provided such training was for completion 
of a training objective initially determined while in a youth 
employment competency system operated under 78% funds. 
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Appendix B defines the mininal structural and procedural elements 
of a sufficiently developed youth employnient competency system as 
well as the minimal requirements for ensuring consistency in the 
reporting of competency attainment in the pre-employment/ work 
maturity skill area. 

Line ?Q Attained Any Competency Area 

Enter in Column C the total unduplicated number of youth termi- 
nees who were enrolled in a youth employment competency component 
and who attained a competency in at least one skill area. 

NOTE: Lines 39-41 are not sub-breakouts of Line 38 because one 
individual may attain several competencies and may be recorded on 
more than one of Lines 39-41. That individual may be recorded 
only once on Line 38, thus, the sum of the entries for Lines 
39-41 must be equal to or greater than the entry for Line 38. 

Line 39 Pr e-Employment/Wor k Maturity Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the pre-employment/wor k maturity skill area. 

Line 40 Basic Education Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the basic education skill area. 

Line 41 Job Specific Skills 

Enter in Column C the number of youth terminees who attained a 
competency in the job specific skill area. 

NOTE: An entry of "O" on any of Lines 39-41 may indicate that 
the PIC has determined that a specific skill area is not 
necessary to become employment competent in their local labor 
mar ket . 
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Follow-up Guidelines 

To ensure consistent data collection and as accurate information 
as possible, procedures used to obtain follow-up information must 
satisfy the following criteria: 

** participant contact should be conducted by telephone or 
in person. Mail questionnaires may be used in those 
cases where an individual does not have a telephone or 
cannot be reached. 

* Participant contact must occur as soon as possible after 
the 13th full calendar week after termination but no 
later than the 17th calendar week after termination. 

* Data reported are to reflect the individual's labor force 
status and earnings during the 13th full calendar week 
after termination and the number of weeks s/he was 
employed throughout the 13-week period after termina'-ion. 

* Interview questions developed by DOL (see following 
Exhibit) must be used to determine the follow-up 
information reported on the JASR. Respondents must be 
told that responding is voluntary and that information 
provided by them will be kept confidential. Other 
questions may be included in the interview. Attitudinal 
questions may precede DOL questions, but questions 
related to employment and earnings must follow. 
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Minimum Postprogram Data Collection Questions 

I want to ask you about the week starting on Sunday, , 

and ending on Saturday, , which was (last 

week/two/three/four weeks ago) . 

1. Did you do any work for pay during th it week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour ii. that week? 

Dollars per hour 

4. How much extra, if any, did you earn in that week from 
tips, overtime, bonupes, commissions, or any work you did 
on the side, before deductions? 

Dollars 



Now I want to ask you about the entire 13 weeks from Sunday. 
, to Saturday, . 

5. Including the week we just talked about, how many weeks 
did you work at all for pay during the 13-week period? 

Weeks [Go to end] 



ALTERNATIVE QUESTIONS 
If answered "NO" to Question 1; 

Now I want to ask you about the entire 13 weeks from Sunday. 
, to Saturday, , ' 

6. Did you do any work for pay during that 13-week period? 

Yes [Go to 7] 

No [Go to end] 

7. How many weeks did you do any work at all for pay durina 
that 13-week period? ^ 
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* Attempts must be made to contact all individuals unless 
tenninee populations are large enough to use sampling. 

" At least six attempts may need to be made to contact 

enough individuals in the follow-up group to obtain the 
required response rate. 

* For each SDA (Title II-A) or combined Statewide (Title 
III Formula and National Reserve) report (JASR), minimum 
response rates of 70% are required for each of the 
following six groups: among adults, those who entered 
employment at termination and those who did not enter 
employment at termination; among welfare recipients, 
those who entered employment at termination and those who 
did not enter employment at termination; and among 
dislocated workers, those who entered employment at 
termination and those who did not enter employment at 
termination. The response rate is calculated as the 
number of terminees with complete follow-up information 
divided by the total number of terminees included in the 
group eligible for follow-up. 

Sampling Procedures 

Where sampling is used to obtain participant contact information. 
It is necessary to have a system which ensures consistent random 
selection of sample participants from all terminees in the group 
requiring follow-up. 

" No participant in the follow-up group may be arbitrarily 
excluded from the sample. 

Procedures used to select the sample must conform to 
generally accepted statistical practice, e.g., a table of 
random numbers or other random selection techniques must 
be used. 

The sample selected for contact must meet minimum sample 
size requirements indicated in Table 1. 

The use of sampling will depend on whether the terrainee 
populations are large enough to provide estimates which meet 
minimum statistical standards. If the number of terminees for 
whom follow-up is required is less than 138, sampling cannot be 
used. In such cases attempts must be made to contact all the 
appropriate terminees. 

Minimum Sample Sizes for Follow-up 

To determine the minimum number of terminees to be included in 
the follow-up sample, refer to Table 1 in the following 
Instructions. Find the row in the left-hand column that contains 
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the planned number of terminees for each of the groups requiring 
follow-up: adults, welfare recipients and dislocated workers. 
The required minimum sample size is given in the middle column of 
that row. The last column gives sampling percentages that will 
assure that the minimum sample size is obtained. 

NOTE: The welfare recipients in the adult sample may be used as 
part of the welfare sample. In this case, an additional number 
of welfare recipients must be randomly selected to provide a 
supplemental sample large enough to meet the same accuracy 
requirements as other groups requiring follow-up. To determine 
the minimum size of this supplemental welfare sample, find the 
row in the left-hand column of Table 1 that contains the planned 
total number of welfare recipients requiring follow-up. From the 
corresponding entry in the middle column, subtract the number of 
welfare recipients included in the adult sample. The remainder 
represents the minimum size of the supplemental sample of welfare 
recipients required for contact. 
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Table 1 

MINIMUM SAMPLE SIZES FOR POLLOW-UP 

Number of Terminees Minimum Sample Sampling 
in Follow-up Population size Percentaq 



1 - 137 


All 


100 


138 - 149 


137 


94 


150 - 159 


143 


92 


160 - 169 


149 


89 


170 - 179 


154 


87 


180 - 189 


159 


85 


190 - 199 


164 


84 


200 - 224 


175 


82 


225 - 249 


185 


78 


250 - 274 


194 


74 


275 - 299 


202 


71 


300 - 349 


217 


67 


350 - 399 


229 


62 


400 - 449 


240 


57 


450 - 499 


250 


53 


500 - 599 




50 


600 - 749 


282 


44 


750 - 999 


302 


38 


1,000 - 1,499 


325 


30 


1,500 - 1,999 


338 


22 


2,000 - 2,999 


352 


17 


3,000 - 4,999 


364 


12 


5,000 or more 


383 


7.3 
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Correcting for Differences in Response Rates 

Different response rates for those terminees who entered employ- 
ment at termination and those who did not are expected to bias 
the performance estimates because those who entered employment at 
termination are more likely to be employed at follow-up. It is 
assumed that those who were employed at termination are easier to 
locate than those who were unemployed because the interviewer has 
more contact sources (e.g., name of employer). The resulting 
response bias can artificially inflate performance results at 
follow-up. 

To account for this problem, separate response rates should be 
calculated for those who were employe<^ at termination and for 
those who were not. These separate response rates should be 
calculated for three groups: all adult II-A terminees, welfare 
recipients and Title III terminees. 

For each group, if the response rates of those employed at 
termination and those not employed differ by more than 5 percen- 
tage points, then the "Worksheet for Adjusting Follow-up 
Performance Measures" in the Follow-up Technical Assistance Guide 
must be used to correct the follow-up measures for that group. 



Appendix B 

PIC-REC0GNI2ED YOUTH EMPLOYMENT COMPETENCIES 

A. General Description of Youth Employment Competency Skill 
Areas 

* Pre-employment skills include world of work awareness, 
labor market knowledge, occupational information, values 
clarification and personal understanding, career 
planning and decision making, and job search techniques 
(resumes, interviews, applications, and follow-up 
letters). They also encompass survival/daily living 
skills such as using the phone, telling time, shopping, 
renting an apartment, opening a bank account, and using 
public transportation; and 

Work maturity skills include positive work habits, 
attitudes, and behavior such as punctuality, regular 
attendance, presenting a neat appearance, getting along 
and working well with others, exhibiting good conduct, 
following instructions and completing tasks, accepting 
constructive criticism from supervisors and co-workers, 
showing initiative and reliability, and assuming the 
responsibilities involved in maintaining a job. This 
category also entails developing motivation and 
adaptability, obtaining effective coping and problem- 
solving skills, and acquiring an improved self image. 

** Basic education skills include reading comprehension, 
math computation, writing, speaking, listening, problem 
solving, reasoning, and the capacity to use these skills 
in the workplace. 

* Job-specific skills — Primary job-specific skills 
encompass the proficiency to perform actual tasks and 
technical functions required by certain occupational 
fields at entry, intermediate or advanced levels. 
Secondary job-spejific skills entail familiarity with and 
use of set-up procedures, safety measures, work-related 
tenninology , recordkeeping and paperwork formats, tools, 
equipment and materials, and breakdown and clean-up 

rout ines. 
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B. Sufficiently Developed Systems for Youth Employment 
Competencies 

A sufficiently developed youth employment competency system must 
include the following structural and procedural elements: 

1 . Quantifiable Learning Objectives 

• PIC-recognized competency statements that are quantifiable, 
employment-related, measurable, verifiable learning 
objectives that specify the proficiency to be achieved as a 
result of program participation. 

Employment competencies/quantifiable learning objectives approved 
by the PIC as relevant to the SDA must include a description of 
the skills/knowledge/attitudes/behavior to be taught, the levels 
of achievement to be attained, and the means of measurement to be 
used to demonstrate competency accomplishment. The level of 
achievement selected should enhance the youth's employabili ty and 
opportunities for postprogram employment. 

2 . Related Curricula, Training Modules, and Approaches 

• Focused curricula, training modules, or behavior 
modification approaches which teach the employment 
competencies in which youth are found to be deficient. 

Such related activities, components, or courses must encompass 
participant orientation, work-site supervisor/instructor/ 
community volunteer training, and staff development endeavors as 
appropriate. They also must include, as appropriate, relevant 
agreements, manuals, implementation packages, instructions, and 
guidelines. A minimum duration of training must be specified 
which allows sufficient time for a youth to achieve those skills 
necessary to attain his/her learning objectives. 

3 . Pre-Assessment 

• Assessment of participant employment competency needs at 
the start of the program to determine if youth require 
assistance and are capable of benefitting from available 
services. 

A minimum level of need must be established before a participant 
is eligible to be tracked as a potential "attained PIC-recognized 
youth employment competency" outcome. All assessment techniques 
must be objective, unbiased and conform to widely accepted meas- 
urement criteria. Measurement methods used must contain clearly 
defined criteria, be field tested for utility, consistency, and 
accuracy, and provide for the training/preparation of all 
raters/scorers . 
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4. Post-Assessment (Evaluation) 

• Evaluation of participant achievement at the end of the 
program to determine if competency-based learning gains 
took place during project enrollment. 

Intermediate checking to track progress is encouraged. All 
evaluation techniques must be objective, unbiased and conform to 
widely accepted evaluation criteria. Measurement methods used 
must contain clearly defined criteria, be field tested for 
utility, consistency, and accuracy, and provide for the 
training/preparation of all raters/scorers. 

5. Employability Development Planning 

• Use of assessment results in assigning a youth to 
appropriate learning activities/sites in the proper 
sequence to promote participant growth and development, 
remedy identified deficiencies, and build upon strengths. 

6. Documentation 

Maintenance of participant records and necessary reporting 
of competency-based outcomes to document intra-program 
learning gains achieved by youth. 

7. Certification 



Proof of youth employment competency attainment in the form 
of a certificate for participants who achieve predetermined 
levels of proficiency to use as evidence of this 
accomplishment and to assist them in entering the labor 
market. 
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C. Guidelines for Ensuring Consistency in the Reporting of 
Pre-Employment/Work Maturity Skill Competencies 

Individuals should demonstrate proficiency in each of the follow- 
ing 11 core competencies. In order for an attainment to be 
reported in the area of pre-employment/wor k maturity, at least 
one PIC-certi f ied competency statement must be developed/ 
quantified in each of the following 11 core competencies — 
provided that at least 5 of these learning objectives were 
achieved during program intervention: 

1. Making Career Decisions 

2 . Using Labor Market Information 

3 . Preparing Resumes 

4 . Filling Out Applications 

5 . Interviewin g, 

6. Being Consistently Punctual 

7. Maintaining Regular Attendance 

8 . Demonstrating Positive Attitudes/Behavior 

9 . Presenting Appropriate Appearance 
Exhibiting Good Interpersonal Relations 
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li. Completing Tasks Effectively 




Appendix C 



Definitions of Terms Necessary 
for Completion of Reports 



EMPLOYMENT/TRAINING SERVICES 

ASSESSMENT — services are designed to initially determine 
each participant's employability , aptitudes, abilities and 
interests, through interviews, testing and counseling to 
achieve the applicant's employment related goals. 

FOLLOW-UP — is the collection of information on a terminee's 
employment situation at a specified period after termination 
from the program, 

INTAKE — includes the screening of an applicant for eligi- 
bility and: (1) a determination of whether the program can 
benefit the individual; (2) an identification of the 
employment and training activities and services which would 
be appropriate for that individual; (3) a determination of 
the availability of an appropriate employment and training 
activity; (4) a decision on selection for participation and 
(5) the dissemination of information on the program. 

OUTREACH activity involves the collection, publication and 
dissemination of information on program services directed 
toward economically disadvantaged and other individuals 
eligible to receive JTPA training and support services. 

YOUTH EMPLOYABILITY ENHANCEMENT TERMINATION 

An outcome for youth, other than entered unsubsidized 
employment, which is recognized as enhancing long-term 
employability and contributing to the potential for long-term 
increase in earnings and employment. The five youth 
employability enhancement outcomes are; 

(1) Demonstrated proficiency in youth employment 
competencies as defined by the PIC in two or more of the 
following three skill areas in which the terminee was 
deficient at enrollment: pre-employment/work maturity, 
basic education, or job-specific skills. During PT '88 
defiSltiSX ^" ^^^^^ area will aeet the 

(2) Entered an occupational-skills employment/training 
program, not funded under Title ll of the JTPA, which 
builds upon and does n ot duplicate training received under 
Ti t le II , 

(3) Returned to full-time school if, at time of intake, 
the participant was not attending school and had not 
obtained a high schooT~d iploma or equivalent, 
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(4) Completed, during enrollment, a level of educational 
achievement which had not been reached at entry. Levels of 
educational attainment are elementary, secondary, and 
post-secondary. Completion standards shall be governed by 
State standards or, at the Governor's discretion, local 
standards at the elementary level; shall include a high 
school diploma, GED Certificate or equivalent at the 
secondary level, and shall require a diploma or other 
written certification of completion at the post-secondary 
level. NOTE: To obtain credit, completion of a major 
level of education must result primarily from participation 
in a JTPA activity. 

(5) Completed program objectives as defined in approved 
exemplary youth project plans if, at time of entry, the 
participant was 14 or 15 years of age. 

EDUCATION STATUS 

SCHOOL DROPOUT — An adult or youth (aged 14-21) who is not 
attending school full-time and has not received a high school 
diploma or a GED certificate. 

STUDENT — An adult or youth (aged 14-21) who has not 
received a high school diploma or GED certificate and is 
enrolled full-time in an elementary, secondary or 
postsecondary-level vocational, technical, or academic 
school or is between school terms and intends to return to 
school . 

HIGH SCHOOL GRADUATE OR EQUIVALENT (No Post-High School) — 
An adult or youth (aged 14-21) who has received a high school 
diploma or GED Certificate, but who has not attended any 
postsecondary vocational, technical, or academic school. 

POST HIGH SCHOOL ATTENDEE — An adult or youth (aged 14-21) 
who has received a high school diploma or GED certificate 
and has attended (or is attending) any postsecondary-level 
vocationax, technical, or academic school. 

FAMILY STATUS 

SINGLE HEAD OF HOUSEHOLD — A single, abandoned, separated, 
divorced or widowed individual who has responsibility for one 
or more dependent children under age 18. 

RACE/ETHNIC GROUP 

WHITE (NOT HISPANIC) — A person having origins in any of the 
original peoples of Europe, North Africa, or the Middle East. 

BLACK (NOT HISPANIC) — A person having origins in any of the 
black racial groups of Africa. 
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HISPANIC — A person of Mexican, Puerto Rican, Cuban, Central 
or South American, or other Spanish culture or origin 
(including Spain), regardless of race. NOTE : Among persons 
from Central and South American countries, only those who are 
of Spanish origin, descent, or culture should be included in 
the Hispanic category. Persons from Brazil, Guiana, and 
Trinidad, for example, would be classified according to their 
race, and would not necessarily be included in the Hispanic 
category. Also, the Portugese should be excluded from the 
Hispanic category and should be classified according to their 
race . 

AMERICAN INDIAN OR ALASKAN NATIVE — A person having origins 
in any of the original peoples of North America, and who 
maintains cultural identification through tribal affiliation 
or community recognition. 

ASIAN OR PACIFIC ISLANDER — A person having origins in any 
of the original peoples of the Far East, Southeast Asia, the 
Indian subcontinent (e.g., India, Pakistan, Bangladesh, Sri 
Lanka, Nepal, Sikkim, and Bhutan), or the Pacific Islands. 
This area includes, for example, China, Japan, Korea, the 
Philippine Islands, ar.J Samoa. Hawaiian natives are to be 
recorded as Asian or Pacific Islanders. 

OTHER BARRIERS TO EMPLOYMENT 

LIMITED ENGLISH LANGUAGE PROFICIENCY — Inability of an 
applicant, whose native language is not English, to 
communicate in English, resulting in a job handicap. 

HANDICAPPED INDIVIDUAL — Refer to Sec. 4(10) of the Act. 
Any individual who has a physical or mental disability which 
for such individual constitutes or results in a substantial 
handicap to employment. NOTE: This definition will be used 
for performance standards purposes, but is not required to be 
used for program eligibility determination (Sec. 4(8) (E)). 

OFFENDER — For reporting purposes, the term "offender" is 
defined as any adult or youth who requires assistance in 
overcoming barriers to employment resulting from a record of 
arrest or conviction (excluding misdemeanors) . 

READING SKILLS BELOW 7th GRADE LEVEL — An adult or youth 
assessed as having English (except in Puerto Rico) reading 
skills below the 7th grade level on a generally accepted 
standardized test. 

NOTE: The following other methods of determination may be 
used : 

° A school record of reading level determined within the 
last 12 months. 

Rev, 9/88 
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** If an applicant is unable to read and therefore cannot 
complete a self -application for the JTPA/EDWAA program, 
s/he may be considered to have English reading skills 
below the 7th-grade level. 

® Individuals with any of the following may be considered 
to have English reading skills above the 7th-grade 
level : 

- A GED certificate received within the last year. 

- A degree (usually a BA or BS ) conferred by a 4-year 
college, university or professional school. 

If there is any question regarding reading ability, a 
standardized test should be administered. 

LONG-TERM AFDC RECIPIENT — An adult or youth listed on the 
welfare grant who had received cash payments under AFDC (SSA 
Title IV) for any 24 or more of the 30 months prior to JTPA 
eligibility determination and who was a welfare recipient (as 
defined below) at the time of such determination. 

U.C. STATUS 

UNEMPLOYMENT COMPENSATION CLAIMANT — Any individual who has 
filed a claim and has been determined monetarily eligible for 
benefit payments under one or more State or Federal 
unemployment compensation programs, and who has not exhausted 
benefit rights or whose benefit year has not ended. 

LABOR FORCE STATUS 

EMPLOYED — (a) An individual who, during the 7 consecutive 
days prior to application to a JTPA program, did any work at 
all: (i) as a paid employee; (ii) in his or her own 
business, profession or farm, or (iii) worked 15 hours or 
more as an unpaid worker in an enterprise operated by a 
member of the family? or (b) an individual who was not 
working, but has a job or business from which he or she was 
temporarily absent because of illness, bad weather, vacation, 
labor-management dispute, or personal reasons, whether or not 
paid by the employer for time off, and whether or not seeking 
another job. (This term includes members of the Armed Forces 
on active duty, who have not been discharged or separated; 
participants in registered apprenticeship programs; and 
self-employed individuals.) 

EMPLOYED PART-TIME ~ An individual who is regularly 
scheduled for work less than 30 hours per week. 
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UNEMPLOYED — An individual who did not work during the 7 
consecutive days prior to application for a JTPA program, who 
made specific efforts to find a job within the past 4 weeks 
prior to application, and who was available for work during 
the 7 consecutive days prior to application (except for 
temporary illness) . 

UNEMPLOYED: 15 OR MORE WEEKS OF PRIOR 26 WEEKS — An 
individual who is unemployed (refer to definition above) at 
the time of eligibility determination and has been unemployed 
for any 15 or more or the 26 weeks immediately prior to such 
determination, has made specific efforts to find a job 
throughout the period of unemployment, and is not classified 
as "Not in Labor Force". 

NOT IN LABOR FORCE — A civilian 14 years of age or over who 
did not work during the 7 consecutive days prior to 
application for a JTPA program and is not classified as 
employed or unemployed. 

WELFARE GRANT INFORMATION 

WELFARE RECIPIENT — An individual listed on the welfare 
grant who was receiving cash payments under AFDC (SSA Title 
IV), General Assistance (State or local government), or the 
Refugee Assistance Act of 1980 (PL 96-212) at the time of 
JTPA eligibility determination. For reporting and 
performance standards purposes, exclude those individuals who 
receive only SSI (SSA Title XVI)"! 

PROGRAM COSTS 

ACCRUED EXPENDITURES — The allowable charges incurred during 
the program year to date requiring provision of funds for: 
(1) goods and other tangible property received; and (2) costs 
of services performed by employees, contractors, 
subrecipients, and other payees. 

NOTE: These charges do not include "resources on order". 
I.e., amounts for contracts, purchase orders and other 
obligations for which goods and/or services have not been 
received . 
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WORKER ADJUSTMENT PROGRAM ANNUAL PROGRAM REPOFT (ETA 9019) 

1. Purpose . The Economic Dislocation and Worker Adjustment 
Assistance Act (EDWAA) Annual Program Report (WAPR) displays 
cumulative data on participation, termination, performance 
measures and the socio-economic characteristics of all 
terminees on an annual basis. The information will be used 
to determine levels of program service and performance 
measures. Selected information will be aggregated to provide 
quantitative program accomplishments on a local. State, and 
national basis. 

2. General Instructions . The Governor will submit: (1) a 
separate WAPR for each designated Substate Area (SSA) ; (2) a 
separate Statewide WAPR summary of the SSA report data (this 
Statewide WAPR summary of individual SSA data should not be 
submitted by single-SSA States); (3) a separate WAPR covering 
participants and terminees in statewide, regional or 
industrywide projects funded under Section 302(c)(1)(B) of 
the Act; and (4) a separate WAPR covering participants and 
terminees in projects funded under Secretary's National 
Reserve Grants (Section 302(a)(2)). Recipients may determine 
whether the reports are submitted on WAPR forms or as a 
computer printout, with data, including signature and title, 
date signed and telephone number, arrayed as indicated on the 
WAPR form. If revisions are made to the WAPR data after the 
reporting deadline, revised copies of the WAPR should be 
submitted to DOL as soon as possible according to the 
required reporting procedures. Submittal of one or more 
WAPRs with revised information for Total Participants, Total 
Terminations and/or Total Program Costs (Federal Funds) 
usually will require submittal of a revised WQFR which 
includes the final quarter of the same program year. 

NOTE: For WAPR reporting purposes, EDWAA shall refer 
to: (1) programs operated by Substate grantees with 
funds authorized under Sections 302(c)(2) and 302(d) or 
otherwise distributed by the Governor under Section 
302(c)(1)(E) and (2) projects operated by the Governor 
with funds authorized under Sections 302(c)(1) and 
302(a) (2) of the Act. 

The reporting period begins on the starting date of each Job 
Training Partnership Act (JTPA) program year, as stated in 
Section 161 of the JTPA. Reports are due in the National and 
Regional Offices no later than 45 days after the end of each 
program year. Two copies of the WAPR are to be provided to: 

U.S. Department of Labor, ETA 
ATTN: TSVR — Room S-5306 
200 Constitution Avenue, N.W. 
Washington, D. C. 20210 

(Rev. 2/90) 



At the same time an additional copy of the WAPR is to be 
provided to the appropriate Regional Administrator for 
Employment and Training in the DOL Regional Office that 
includes the State in which the JTPA recipient is located. 

NOTE: The current JASR, ETA 8580 (June 1988), is to continue 
to be used for programs operating solely with PY88 and 
earlier year's Title III funds. The WAPR is to be used 
beginning July 1, 1989, for °Y89 programs funded under EDWAA 
(including carryin funds used for EDWAA purposes, if any). 

Facsimile of Form . See the following page. 

Instructions for Completing the Worker Adjustment Program 
Annual Program Report (WAPR) . 

a. State/Substate Area Name and Address 

On separate Section 302(c)(1) and Section 302(a)(2) 
reports, enter the name and address of the State agency 
that will administer the Statewide programs. For SSA 
reports, enter the name and address of the Substate 
grantee that will administer the sSA programs. 

b. Report Type 

Designate the type of report data provided on this WAPR. 
If this is an SSA report, also enter the ETA-assigned 
Substate Area Code number. 

c . Report Period 

Enter in "From" the beginning date of the designated JTPA 
program year and enter in "To" the ending date of that 
program year . 

d. Signature and Title (at bottom of the page) 

The authorized official signs here and enters his/her 
title. 

e. Date signed 

Enter the date the report was signed by the authorized 
official. 

f . Telephone Number 

Enter the area code and telephone number of the 
authorized official. 
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Annual Program Report (WAPR; 



U.S. Depa'tnen: o* ^£Dor 

Empioyme'^: ana Training Aominisirauon 



a. suitt/substite Area Name and Address 





b. ^apoHType 

( ) SSA # 

( ) GovStaiawida 
( ) SacyN/Rasv 

c. Repon Period 



0MB No. 120M274 
Expires 12/31/92 



From: 7/1/19 



To: 6/30/19 



I. PARTICIPATION AND TERMINATION SUMMARY 



A. ittued Certificate of Continuing Eligibility (CCE) 



Corwurrent 
ParticiDants 



B. 



All 

Participants 



B. TOTAL PARTICIPANTS 



1. All CCEs Reaeenr)ed fo r Retraining 
C. TOTAL TERMINATIONS 



1. Entered Unsubsidized Enfipioyment From Retraining 



a. Relocated Out of Area 



2. Entered Unsubttdized Employment From Basic Readjustment Services ONLY 



3. Called BacK/Rematned with the Layoff Employer 



4. Transferred to Other JTPA Programs 



5. Entered Non.J7PA Training 



6, All Other Terminations 
1 1| II. TERMINEE CHARACTERISTICS AND PERFORMANCE MEASURES SUMMARY 




1^1 S<nple Heed of Houethold With DepefH<ent(i) Under Ape 18 



12 
13 
14 
15 
16 
17 



Black (Not Hispanic) 



Hispanic 



Amer teen indiar. or Alas kan Native 
Asian Of Pacific lelander 
Limlfd English LanQuape Proffcitncy 

Handicapped 

Riadino Skills Bek>w 7th4kade Level 




BEST COPY AVAILABi 



Report Period 



From 
7/1/19 



To 

6/X/19 




II. TERMINEE CHARACTERISTICS AND PERFORMANCE luiEASURES SUMMARY - CONTINUED 



All Tflrmir)tet 



20 


1)1 


U.C Clalnr\ant 




21 




Unemployed 15 or More Weeks of Prior 26 Weeks 


22 
23 


ss 


Veteran (Total) 


VlatnarrwEra i 




24 


Average Weeks Panicipated 




25 


Average Hourly Wage - Pre-Program 


! 


26 


1 Average Hourly Wage at Termir^ation j 


27 


1 Total Program Costs (Federal Funds) 


26 1 N«eds*Raiat0d Piymenu 




29 Total Availabit Faderai Funds 




111. FOLLOW-UP INFOI^MATION 


X 


Employment Rata at Follow-up | 


31 


Average Hourly Wage at Follow-up j 


32 


Average Number of Weeks Worked In Follow-up Period j 


33 


Sample Size | 


34 


^sponse Rata 






IV. RETRAININQ/BA8IC READJUSTMENT SERVICES 


35 


Received Basic Readjustment Services ONLY 




36 


Received ANY Retraining Activity 




37 


Uss than 26 Weeks 






36 


26 or More Weeks 






39 


Connpiettd Daearoom Training: Basic Education or Attained QED 




40 


Completed Orvth»-Job Training 




41 


Compltttd Othr Occupational Skills Training 






Pagt 2 of 2 Pagt* 



(March 1868) 




ERIC 



-3- 




^' General Inf orma tioii . Unless otherwise indicated/ data 

reported on characteristics of terminees should be based on 
information collected at the time of eligibility 
determination , 



CHARACTERISTICS INFORMATION OBTAINED ON AN INDIVIDUAL AT THE TIME 
OF ELIGIBILITY DETERMINATION FOR THE RECIPIENT'S EDWAA PROGRAM 
SHOULD NOT BE UPDATED WHRN THE INDIVIDUAL TERMINATES FROM THE 
EDWAA PROGRAM. 




NOTE: Recipients shall ensure that individuals are enrolled 
within 45 days of the date of application or a new application 
must be taki n (20 CFR 629.1). This 45-day period for Certificate 
of Continuing Elibibility (CCE) holders should begin upon 
completion of a full participant record, usually at the time of 
CCE redemption, as this may be at a location other than where the 
CCE was issued and such information taken earlier to determine 
eligibility may not be readily available at the redemption site. 

SECTION I - PARTICIPATION AN D TERMINATION SUMMARY 

Section I displays the EDWAA program's accomplishments in terms 
of the total cumulative number of participants in the program, 
the number and types of terminations from the program, and ti.e 
number of CCEs issued/redeemed, as of the end of the reporting 
period. ^ ^ 

In Section I, Column A., Item I.B., enter individuals who are 
concurrent participants and are receiving Basic Readjustment 
Services and/or Retraining under another EDWAA-funded 
grantee/program, JTPA title or have entered non-JTPA/EDWAA 
training for the completion of the initially determined training 
objective, prior to termination from their initial EDWAA 
program. Column A. is a sub-breakout of Column B. for this line 
1 tem • 



In Section I, Column A., Item I.e., enter terminees who have been 
concurrent participants and have received Basic Readjustment 
Services and/or Retraining under another EDWAA-funded 
' grantee/program, JTPA title or have received non-JTPA/EDWAA 
training for the completion of the initially determined training 
objective, prior to termination from their initial EDWAA 
program. Also, distribute these terminees between Items I.C.I, 
and I.e. 2., as appropriate, on the basis of the final type'of 
termination from the EDWAA program of final participation. 
Column A. is a sub-breakout of Column B. for these line items. 
NOTE: An individual included in a line item entry in Column A 
also must be included in the entry for the same line item in 
Column B. 



Entries for Items I. A, I.B. and I.e. are cumulative from the 
beginning of the program year through the end of the reporting 
period. 
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Item I. A. Issued Certificate of Continuing Eligibility (CCE) 

Enter the total number of applicants to whom a Certificate of 
Continuing Eligibility was issued during this program year, as 
provided in Section 316(b) of the Act, by the Substate grantee 
through the end of the reporting period. NOTE: Enter zero for 
this item on the Governor's Statewide report (Section 302(c)(1)). 

Item I.B. Total Participants 

Enter by column the total number of participants who are or were 
receiving employment, training or services (except post-termina- 
tion services) through the end of the reporting period, including 
both those on board at the beginning of the designated program 
year and those who have entered during the program year. 

"Participant" means any individual who has: (1) been determined 
eligible for participation upon intake; and (2) started receiving 
employment, training, or services (except post-termination 
services) funded under the Act, following intake. Individuals 
who receive only outreach and/or intake and initial assessment 
services or postprogram follow-up are excluded. 

NOTE: Also exclude individuals who receive only Rapid Response 
Assistance and information, per Section 314(b), provided by the 
State's Dislocated Worker Unit. 

If individuals receive concurrent employment, training and/or 
services under more than one title/program, they are to be 
considered participants in both titles/programs for purposes of 
recording actual number of weeks participated, dollars expended, 
and other pertinent data. Individuals who initially participate 
in EDWAA funded activity and subsequently participate in any 
other EDWAA (or non-EDWAA) funded activity, FOR THE COMPLETION OF 
THE INITIALLY DETERMINED TRAINING OBJECTIVE, may be considered to 
be concurrent participants in each program. 

The sum of the entries (all SSAs in a State) in Item I.E., Total 
Participants, and in Item I.C., Total Terminations, of the WAPR 
should equal the entries for Substate Grantees in Column B., 
Lines 16 and 17, respectively, of the WQFR for the final quarter 
of the same program year for the same recipient. 

The entries in Item I.B. and in Item I.C. of the Governor's 
Statewide WAPR should equal the entries for Governor's Reserve in 
Column A., Lines 16 and 17, respectively, of the WQFR for the 
final quarter of the same program year for the same recipient. 

The entries in Item I.B. and in Item I.C. of the Secretary's 
National Reserve WAPR should equal the entries for the 
Secretary's National Reserve in Column C, Lines 16 and 17, 
respectively of the WQFR, for the final quarter of the same 
program year for the same recipient. 
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Item I.B.l. All CCES Redeemed for Retraining 

Enter the total number of unexpired participant CCEs, regardless 
of year issued, redeemed for Retraining during this program year 
by the Substate grantee, through the end of the reporting 
period. Include all CCEs so redeemed that were issued by any 
Substate grantee for periods not to exceed 104 weeks prior to 
redemption. This item is a sub-breakout of Item I.B. 

Item I.e. Total Terminations 

Enter by column the total number of participants who terminated 
(as defined below) from the program during the reporting period. 
Include all participants who received no Basic Readjustment 
Services (except supportive services and/or counseling) or 
Retraining for 90 days. This iten is the sua of I tens I.C.I, 
through I.C.6 . — — . 

"Termination" means the separation of a participant from the pro- 
gram who is no longer receiving Basic Readjustment Services or 
Retraining under EDWAA. Individuals may be considered 
participants for up to 90 days after last receipt of Basic 
Readjustment Services or Retraining, during which time they may 
continue to receive supportive services, as provided for in 
Section 314 (c)(15) and Section 4(24) of the^Act. 

For purposes of calculating average weeks participated, this 
single period of up to 90 days between "last receipt of Basic 
Readjustment Services or Retraining under EDWAA" and actual date 
of termination is defined as "inactive status" and is not to be 
included in "Average Weeks Participated". Terminees may continue 
to receive counseling necessary to assist in the retention of 
employment, for not more than 6 months following last receipt of 
Basic Readjustment Services (Section 314 (c)) or Retraining 
(Section 314 (d) ) . 

Participants who have transferred from one title to another, or 
between programs of the same title, should be recorded as 
terminations from the title or program of initial participation 
and included as participants in the title or program into which 
they have transferred. If they are concurrent participants in 
more than one title or program, the type of termination 
determined for the fina) program should be recorded for all 
programs for these participants. 

For purposes of calculating average weeks participated for such 
concurrent EDWAA program participants, the period between "last 
receipt of Basic Readjustment Services and Retraining funded 
under a given EDWAA program" (i.e., SSA, Governor's Reserve or 
Secretary's National Reserve) and actual date of termination from 
that EDWAA program is defined as "inactive status" and is not to 
be included in Line 24. 
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Item I.C.I. Entered Unsubsidized Employment from Retraining 

Enter by column the total number of participants who, at 
termination, entered full- or part-time unsubsidized employment 
from Retraining through the end of the reporting period. (These 
participants may or may not have received Basic Readjustment 
Services.) This item is a sub-breakout of Item I.e. 

NOTE: Unsubsidized employment means employment not financed from 
funds provided under the Act and includes, for JTPA/EDWAA 
reporting purposes, entry into the Armed Forces of the U. S., 
entry into employment in a registered apprenticeship program and 
terminees who become self-employed. 

Item I.C.I. a. Relocated Out of Area 

Enter the total number of participants who, at termination from 
other Retraining, entered unsubsidized employment after receiving 
relocation assistance and relocating outside the Substate Area 
which provided such relocation assistance, or within or outside 
of the State, if this assistance was provided by a Statewide SSA 
or by a program administered by the Governor. This item is a 
sub-breakout of Item l.C.l. 

Item I.e. 2. Entered Unsubsidized Employment From Basic 
Readjustment Services ONLY 

Enter by column the total number of participants who, at 
termination, entered full- or part-time unsubsidized employment 
from Basic Readjustment Services ONLY through the end of the 
reporting period. This item is a sub-breakout of Item l.C. 

Item I.e. 3. ealled Back/Remained With the Layoff Employer 

Enter the number of terminees from the EDWAA program who, after 
being laid off by an employer, were recalled by that employer to 
a permanent job at the same or another location. Also include 
EDWAA program termineea who remained in a permanent job with an 
employer after receipt of a layoff notice from that employer. 
This item is a sub-breakout of Item I.e. NOTE: Do not include 
such terminees in the entry for Items I.e.l. or I.e. 2., above. 

Item I.e. 4. Transferred to Other JTPA Programs 

Enter the number of terminees who transferred to and entered 
programs funded under another JTPA title including Title 
Ill-Formula and Title Ill-National Reserve. Also include on this 
line terminees vjuo transferred to and entered EDWAA programs 
operated by another Substate grantee, or who transferred to and 
entered EDWAA programs operated by the Governor (and 
conversely) . This item is a sub-breakout of item l.C. 

(Rev. 2/90) 
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Item I.e. 5. Entered Non-JTPA Training 

Enter the number of terminees who entered, during the program 
year, training not funded with JTPA monies. This item is a 
sub-breakout of Item I.C. 

Item I.C. 6. All Other Terminations 



Enter by column the total number of participants who were termi- 
nated for reasons other than those in Items I.C.I, through 
I.C. 5., successful or otherwise, through the end of the reporting 
per iod . 

SECTION II - TERMINEE PERFORMANCE MEASURES INFORMATION 

Section II displays performance measures/parameters information. 
As indicated previously, data reported on characteristics of 
terminees should be based on information collected at time of 
eligibility determination unless otherwise indicated. 

Governors may develop any participant record which meets the 
requirements of Section 629.35(c) and (d) of the JTPA 
regulations. The DOL/ETA Technical Assistance Guide ; The JTPA 
Participant Record , dated May 1983, may be used as a refeTence . 

Line Item Definitions and Instructions 

Sex 

Line 1 Male 
Line 2 Female 

Distribute the terminees according to Sex. The sum of Lines 1 
and 2 should equal Item I.C. 



Age 

Line 3 29 and Under 

Line 4 30-44 

Line 5 45-54 

Line 6 55 and Over 



Distribute the terminees according to Age. The sum of Lines 3 
through 6 should equal Item I.C. 

Education Status 



Line 7 
Line 8 
Line 9 
Line 10 



Less Than High School 

High School Graduate or Equivalent (No Post-High School) 
Post-High School At .indee 

College Graduate and Above 



Distribute the terminees according to Education Status. The sum 
of Lines 7 through 9 should equal Item I.C. NOTE: Line 10 is a 
sub-breakout for a specific group included in Line 9. 
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Family Sta tus 

Line 11 ^^"5^® °^ Household with Dependent(s) Under Age 

Enter the total number of terminees for whom the above Family 
Status classification applies. 

Race /E thnic Gro up 

Line 12 White (Not Hispanic) 
Line 13 Black (Not Hispanic) 
Line 14 Hispanic 

Line 15 American Indian or Alaskan Native 
Line 16 Asian or Pacific Islander 

Distribute the terminees according to the Race/Ethnic Groups 
listed above. For purposes of this report, Hawaiian Natives are 

12 throuoT?^ ^ "'""'^^^ Islander". The sum of Lines 

12 through 16 should equal Item I.e. 

Other Ba rriers to E mployment 

Line 17 Limited English Language Proficiency 
Line 18 Handicapped 

Line 19 Reading Skills Below 7th Grade Level 

o?^!r n^^ total number of terminees for whom each of the above 
Other Barriors to Employment apply. 

§ii2i£ its_S t at u s 

Line 20 U. c. (Unemployment Compensation) Claimant 

hr!!^5^^ "If"^^'' °^ terminees for whom each of the above 

benefits status classifications apply. 

L abor Force Status 

Line 21 Unemployed: 15 or More Weeks of Prior 26 Weeks 

po^^I °^ terminees for whom the above Labor 

Force Status classification applies. 

Vete r an Status 

Line 22 Veteran (Total) 
Line 23 Vietnam Era 

Enter the total number of terminees for whom each of the above 
ll^^'in'l ^J^ff^fications apply, as defined in siTtion 4 (26) (A) (B) 
and (D of the Act. Line 23 i.' a sub-breakout for a specif c 
group included in Line 22. oFe^ii-ic 
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Qtjlgg_ Prog rani Information 

Line 24 Average Weeks Participated 

Enter the average number of weeks of participation in the EDWAA 
program for all terminees. Weeks of participation include the 
period from the date an individual becomes a participant in EDWAA 
through the date of a participant's last receipt of Basic 
Readjustment and/or Retraining. Exclude the single period of up 
to 90 days during which an individual may remain in an inactive 
status prior to termination. Time in inactive status for all 
terminees should not be counted toward the actual number of weeks 
participated. Inactive status is defined as that period between 
"last receipt of Basic Readjustment Services and/or Retraining 
under EDWAA" and actual date of termination. (See Item I.e.) 

To calculate this entry: Count the number of days participated 
for each terminee/ including weekends, from the start date of 
his/her participation in EDWAA until his/her last receipt of 
Basic Readjustment Services and/or Retraining under EDWAA. 
Divide this result by 7. This will give the number of weeks 
participated for that terminee. Sum all the terminees' weeks of 
participation and divide the result by the number of terminees, 
as entered in Item I.e. This entry should be reported to the 
nearest whole week. 



Line 25 Average Hourly Wage — Pre-Program 

Enter the average hourly pre-EDWAA wage of all terminees. In 
calculating this average, use the hourly wage from the job of 
dislocation. Those terminees who had no pre-EDWAA employment 
should be counted as "$0.00" hourly wage. 

To calculate this entry: Sum the pre-program hourly wage for all 
terminees shown in Item i.e. Divide the result by the number of 
terminees shown in item i.e. NOTE: For the calculation, use 
the hourly wage regardless of whether the individual was employed 
fu^l- or part-time. ' 

Line 26 Average Hourly Wage at Termination 

Enter the average hourly wage at termination for the total number 
of terminees in Items I.C.l. through I.e. 3. 

To calculate this entry: Sum the hourly wage at termination for 
all the terminees shown in Items I.e.l. through I.e. 3. Divide 
the result by the number of terminees shown in Items I e 1 
through I.e. 3. 

Hourly wage includes any bonuses, tips, gratuities and 
coDDissions earned. 
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Line 27 Total Program Costs (Federal Funds) 

Enter the total accrued expenditures, through the end of the 
reporting period, of the funds allocated to the SSA under 
Sections 302(c)(2) and 302(d) of the Act or otherwise distributed 
by the Governor to the SSAs under Section 302(c)(1)(E). On the 
separate WAPRs : (1) covering participants and terminees projects 
funded under Section 302(c)(1); (2) or participants and terminees 
in Secretary's National Reserve Grants projects funded under 
Section 302(a)(2) of the Act, enter the total expendidures for 
all participants and terminees served in such programs through 
the end of the reporting period. Include, as appropriate, 
accrued expenditures against JTPA Title III funds provided for 
PY88 and PY87 which were carried into EDWAA and used for EDWAA 
purposes, if any. Include expenditures of Federal funds only. 

NOTE: Entries will be nade to the nearest dollar. Negative 
entries are not acceptable. The WAPR program cost data will be 
compiled on an accrual basis. If the recipient's accounting 
records are not normally maintained on an accrual basis, the 
accrual information should be developed through an analysis of 
the records on hand or on the basis of best estimates. 

The sum of the entries for Line 27, Total Program Costs, of the 
WAPR (i.e., total for the State's SSAs under EDWAA) should equal 
the entry for Column B., Line 10, Total Accrued Expenditures 
(Substate Grantees) of the WQFR for the same recipient that 
includes the final quarter of the same program year. 

Line 27 of the Statewide WAPR for the Governor's Reserve activity 
(Section 302(c)(1)) should equal Column A., Line 10 of the WQFR 
for the same recipient that includes the final quarter of the 
same program year; and Line 27 on the WAPR for the Secretary's 
National Reserve Grants (Section 302(a)(2)) to the state should 
equal Column B., Line 5 on that WQFR. 

Line 28 Needs-Related Payments 

Enter the total accrued expenditures for needs-related payments 
to eligible dislocated workers who do not qualify or have ceased 
to qualify for Unemployment Compensation, in order to enable such 
workers to participate in training or education programs under 
EDWAA (Section 314(e)). This is a sub-breakout of Line 27. 

Line 29 Total Available Federal Funds 

Enter the total Federal funds available for the EDWAA program 
described on this report including (1) unexpended funds carried 
over from previous program years, (2) funds allocated or awarded 
for this program year, and (3) any reallocation that increased or 
decreased the amount of funds available for expenditure through 
the end of this reporting period. Entries will be made to the 
nearest dollar . 



ERIC 



1^7 



-11- 



The sum of the entries for Line 29, Total Available Federal 
Funds, of the WAPR (i.e., total for the State's SSAS under EDWAA) 
should equal the entry for Column B., Line 9, Total Federal Funds 
Available (Substate Grantees) of the WQFR for the same recipient 
that includes the final quarter of the same program year. 

Line 29 of the Statewide WAPR for the Governor's Reserve activity 
(Section 302(c)(1)) should equal Column A., Line 9 of the WQFR 
for the same recipient that includes the final quarter of the 
same program year; and Line 29 on the WAPR for the Secretary's 
National Reserve Grants (Section 302(a)(2)) to the State should 
equal Column B., Line 1 on that WQFR. 

SECTION III - FOLLOW-UP INFORMATION 

Section III displays information based on follow-up data which 
must be collected through participant contact to determine an 

^°^?^ ^^^^""^ earnings, if any, during the 
13th full calendar week after termination and the number of weeks 
s/he was employed during the 13-week period. Follow-up data 
should be collected from participants whose 13th full calendar 
week after termination ends during the program year (the 
follow-up group). Thus, follow-up will be conducted for 
individuals who terminate during the first three quarters of the 
program year and the last quarter of the previous program year. 

For PY89 follow-up may be conducted for individuals who terminate 
during the first three quarters of the program year and 
postprogram data collection need not begin until October 1, 1989. 

In order to ensure consistency of data collection and to 
guarantee the quality of the follow-up information, follow-up 
procedures must satisfy certain criteria. (See the Follow-up 
Guidelines included in these WAPR instructions. Appendix A.) 

Other procedures used to collect the follow-up data are at the 
discretion of the Governors. 

NOTE: Every precaution must be taken to prevent a "response 
bias which could arise becaure it may be easier to contact par- 
ticipants who were employed at termination than those who were 

entered employment at termination are 
more likely to be employed at follow-up. Special ptocedures have 
been developed by which sSAs and States can monitor response 
bias. If your response rates for those who were and were not 
employed at termination differ by more than 5 percentage points, 
the follow-up entries for the WAPR must be calculated using the 
worksheet for AdDUSting Follow-up Performance Measures" in the 
Follow-up Technical Assistance Guide. If the respon se rates 
differ by 5 percentage points or le 3, the foliowiSg instructions 
for completing Lines 30-34 may be t d . 

Line 30 Employment Rate at Follow-up 

Enter the employment rate at follow-up. 
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Calculate the employment rate by dividing the total number of 
respondents who were employed (full-time or part-time) during the 
13th full calendar week after termination by the total number of 
respondents (i.e., terminees who completed follow-up 
interviews). Then multiply the result by 100. This entry should 
be reported to the nearest one decimal (00.0). 

Line 31 Average Hourly Wage at Follow-up 

Enter the average hourly wage of those employed (full-time or 
part-time) at follow-up. 

To calculate this entry: Sum the hourly wage (and, if 
appropriate, add tips, overtime, bonuses, etc.) of each 
respondent employed at follow-up. Divide the sum of hourly wage 
for all respondents employed during the 13th full calendar week 
after termination by the number of respondents employed at the 
time of follow-up. Respondents not employed at follow-up are not 
included in this average. 

Include any wages, bonuses, tips, gratuities, commissions and 
overtime pay earned. 

Line 32 Average Number of Weeks Worked in Follow-up Period 

Enter the average number of weeks worked in follow-up period. 

To calculate the average number of weeks worked (full-time or 
part-time), divide the sum of the number of weeks worked during 
the 13 full calendar weeks after termination for all respondents 
who worked, by the total number of all respondents, whether or 
not they worked any time during this 13-week follow-up period. 
This entry should be reported to the nearest one decimal (00.0). 

Line 33 Sample Size 

Enter by column the size of the actual sample selected to be con- 
tacted for follow-up. 

NOTE: If oversampling was used, the sample size should include 
all those selected, not just the required minimum sample size. 
Those deceased or severely incapacitated to the point of being 
unable to respond at follow-up may be excluded from the sample 
Size. ^ 

Line 34 Response Rate 

Enter the overall response rate, i.e., the percentage of complete 
surveys obtained. v,uiiit.j.cte 

To calculate the overall response rate, divide the number of 
terminees with complete follow-up information by the total number 
of terminees included in the follow-up sample (Line 33) and 
multiply by 100. This entry should be reported to the nearest 
whole percent. 
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NOTE: Complete follow-up information consists of substantive 
answers to the required follow-up questions and may not include 
"don't know", "no answer" or "don't remember". 

SECTION IV - RETRAINING/BASIC READJUSTM^'^NT SERVICES 

Section IV displays information relevant to program activities 
funded under EDWAA. 

Line 35 Received Basic Readjustment Services ONLY 

Enter the total number of terminees, regardless of type of 
termination, who received Basic Readjustment Services ONLY, as 
indicated in Section 314(c) of the Act. The sum of Lines 35 and 
36 should equal Item I.C. NOTE: Individuals who receive only 
outreach and/or intake and initial assessnent services or Rapid 
Response assistance are not participants/terminees. 

Line 36 Received ANY Retraining Activity 
Line 37 Less than 26 Weeks 

Line 38 26 or More Weeks 

Enter the total number of terminees, regardless of type of 
termination, who received ANY Retraining activity included in 
Section 314(d) of the Act. Lines 37 and 38 are sub-breakouts of 
Line 36 and should be used to distribute terminees who received 
ANY Retraining activity by actual length of stay in all 
Retraining activities, whether or not such Retraining was 
completed. These terminees may or may not have received Basic 
Readjustment Services. The sum of Lines 37 and 38 should equal 
Line 36. 

Note: Terminees who have received retraining activity funded 
under a cooperative agreement with : (1) other JTPA monies 
(i.e. 3%, 8%, Title II etc.) or (2) other than JTPA funds may be 
counted in Lines 36-41, PROVIDED SOCH TRAINING WAS FOR THE 
COMPLETION OP THE INITIALLY DETERMINED TRAINING OBJECTIVE. 

Line 39 Completed Classroom Training: Basic Education or 

Attained GED 
Line 40 Completed On-the-Job Training 
Line 41 Completed Other Occupational Skills Training 

Enter the total number of terminees for which each of these 
Retraining completion/attainment classifications apply. A 
terminee should be included in all appropriate categories. 

NOTE: Basic Education in Line 39 includes remedial reading, 
writing, mathematics and/or English for non-English speakers. 
Attainment of a GED or a high school diploma upon completion of 
any training also should be included. (A terminee may be counted 
only once in this line item, as appropriate.) 
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Appendix A 



Follow-up Guidelines 



To ensure consistent data collection and as accurate information 
as possible, procedures used to obtain follow-up information must 
satisfy the following criteria: 

* Participant contact should be conducted by telephone or 
in person. Mail questionnaires may be used in those 
cases where an individual does not have a telephone or 
cannot be reached . 



Participant contact must occur as soon as possible after 
the 13th full calendar week after termination but no 
later than the 17th calendar week after termination. 

Data reported are to reflect the individual's labor force 
status and earnings during the 13th full calendar week 
after termination and the number of weeks s/he was 
employed throughout the 13-week period after termination. 

Interview questions developed by DOL (see following 
Exhibit) must be used to determine the follow-up 
information reported on the WAPR. Respondents must be 
told that responding is voluntary and that information 
provided by them will be kept confidential. Other 
questions may be included in the interview. Attitudinal 
questions may precede DOL questions, but questions 
related to employment and earnings must follow. 

Attempts must be made to contact al_l individuals unless 
terminee populations are large enough to use sampling. 

At least six attempts may need to be made to contact 
ri.ough individuals in the follow-up group to obtain the 
required response rate. 

For each SSA and for Statewide and National Reserve 
reports (WAPR) , minimum response rates of 70% are 
required for each of the following two groups •>£ 
dislocated workers: those who entered employment at 
termination and those who did not enter employment at 
termination. The response rate is calculated as the 
number of terminees with complete follow-up information 
divided by the total number of terminees included in the 
group eligible for follow~up. 
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EXHIBIT 

Minimum Postprograin Data Collection Questions 

I want to ask you about the week starting on Sunday, , 

and ending on Saturday, , which was (last 

week/two/three/four weeks ago) . 

1. Did you do any work for pay during that week? 

Yes [Go to 2] 

No [Go to C] 

2. How many hours did you work in that week? 

Hours 

3. How much did you get paid per hour in that week? 

Dollars per hour 

4. HOW much extra, if any, did you earn in that week from 
tips, overtime, bonuses, commissions, or any work you did 
on the side, before deductions? 

Dollars 



Now I want to ask you about the entire 13 weeks from Sunday, 
, to Saturday, . 

5. Including the week we just talked about, how many weeks 
did you work at all for pay during the 13-week period? 

Weeks [Go to end] 



ALTERNATIVE QUESTIONS 
If answered "NO" to Queotion 1: 

Now I want to ask you about the entire 13 weeks from Sunday, 
t to Saturday, . 

6. Did you do any work for pay during that 13-week period? 

Yes [Go to 7] 

No (Go to end] 

7. How many weeks did you do any work at all for pay during 
that 13-week period? 
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T erminee Pop u lat io ns for F ollow-up 

Each program (SSA> Governor's Reserve/ Secretary's National 
Reserve) in EDWAA is responsible for conducting a follow-up of 
all or a sample of parti-'ipants who have terminated from that 
program. 

The ^universe" of terminees for the follow-up includes all 
participants who terminated from a program. Those participants 
who may have been concurrent participants in more than one JTPA 
title/EDWAA program will be in the universe for each. 

When selecting a sample from the universe, each title/program 
will be treated separately, so that an individual who had been a 
concurrent participant might be selected in one sample but not 
another. This, however, does not preclude the possibility that 
the participant might be selected in more than one sample. In 
the event that a concurrent participant has been selected in more 
than one sample, the responses collected from a single interview 
may be shared among the different titles/programs to avoid the 
necessity of multiple interviews with the same individual. 

When an individual who has terminated from one title/program and 
subsequently become a participant in another program (i.e., not a 
concurrent participant) is selected in the sample for both 
titles/programs, separate interviews must be conducted. Further, 
if an individual is selcted in one sample and is a participant in 
another title/program at the time of the inverview, regular 
follow-up information should be determined and recorded. 

Sampling Proce d ures 

Where sampling is used to obtain participant contact information. 

It is necessary to have a system which ensures consistent random 

selection of sample participants from all terminees in the qrouo 

requiring follow-up. ^ 

No participant in the follow-up group may be arbitrarily 
excluded from the sample. 

Procedures used to select the sample must conform to 
generally accepted statistical practice, e.g., a table of 
random numbers or other random selection techniques must 
be used. 





The sample selected for contact must meet minimum sample 
size requirements indicated in Table 1. 

The use of sampling will depend on whether the terminee 
populations are large enough to provide estimates which meet 
minimum statistical standards. If the number of terminees for 
whom follow-up is required is less than 138, sampling cannot be 
used. In such cases attempts must be made to contact all the 
appropriate terminees. 
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Table 1 

MINIMUM SAMPLE SIZES FOR FOLLOW-UP 

Number of Terminees Minimum Sample Sampling 

in_Fql low-up Po pul ation_ size Percent ages 

1 - 137 All 100 

138 - 149 137 94 

150 - 159 143 92 

160 - 169 149 89 

170 - 179 154 87 

180 - 189 159 85 

190 - 199 164 84 

200 - 224 175 82 

225 - 249 185 78 

250 - 274 194 

275 - 299 202 71 

300 - 349 217 67 

350 - 399 229 62 

400 - 449 240 57 

450 - 499 250 53 

500 - 599 265 50 

600 - 749 282 44 

750 - 999 302 38 

1,000 - 1,499 325 30 

1/500 - 1,999 338 22 

2,000 - 2,999 352 17 

3,000 - 4,999 364 12 

5,000 or more 383 7,3 
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Mi n imum Sampl e Size s for F o 1 low- up 

To determine the minimum number of terminees to be included in 
the follow-up sample/ refer to Table 1 in the following 
instructions. Find the row in the left-hand column that contains 
the planned number of dislocated worker terminees. The required 
minimum sample size is given in the middle column of that row. 
The last column gives sampling percentages that will assure that 
the minimum sample size is obtained. 

C orrecting for Difference s in Responae_Ra tes 

Different response rates for those terminees who entered employ- 
ment at termination and those who did not are expected to bias 
the performance estimates because those who entered employment at 
termination are more likely to be employed at follow-up. It is 
assumed that those who were employed at termination are easier to 
locate than those who were unemployed because the interviewer has 
more contact sources (e.g./ name of employer). The resulting 
response bias can artificially inflate pjajrf ormance results at 
follow-up. 

To account for this problem, separate response rates should be 
calculated for those who were employed at termination and for 
those who were not. 

If the response rates of those employed at termination and those 
not employed differ by mere than 5 percentage points, then the 
"Worksheet for Adjusting Follow-up Performance Measures" in the 
Follow-up Technical Assistance Guide must be used to correct the 
follow-up measures. 
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Definitions of Terms Necessary 
for Completion of Reports 



EMPLOYMENT/TRAINING SERVICES 

ASSESSMENT — services are designed to initially determine 
each participant's employabi li ty , aptitudes, abilities and 
interests, through interviews, testing and counseling to 
achieve the applicant's employment related goals. 

FOLLOW-UP — IS the collection of information on a terminee's 
employment situation at a specified period after termination 
from the program. 

INTAKE — includes the screening of an applicant for eligi- 
bility and: (1) a determination of whether the program can 
benefit the individual; (2) an identification of the 
employment and training activities and services which would 
be appropriate for that individual; (3) a determination of 
the availability of an appropriate employment and training 
activity; (4) a decision on selection for participation and 
(5) the dissemination of information on the program, 

OUTREACH — activity involves the collection, publication and 
dissemination of information on program services directed 
toward economically disadvantaged and other individuals 
eligible to receive JTPA training and support services. 

EDUCATION STATUS 

LESS THAN HIGH SCHOOL — An adult or yoi;hh- (1) who is not 
attending school full-time and has not received a high school 
diploma or a GED certificate; or (2) who has not received a 
high school diploma or GED certificate and is enrolled 
full- tine in an elementary, secondary or postsecondary-level 
vocational, technical, or acadeaic school or is between 
school terms and intends to return to school. 

HIGH SCHOOL GRADUATE OR EQUIVALENT (No Post-High School) — 
An adult or youth who has received a high school diploma or 
GED Certificate, but who has not attended any postsecondary 
vocational, technical, or academic school. 

POST HIGH SCHOOL ATTEMOEE — An adult or youth who has 
received a high school ploma or GED certificate and has 
attended (or is attendiiig) any postsecondary-level 
vocational, technical, or academic school. 
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COLLEGE GRADUATE — A terminee who has received a degree 
(usually a BA or BS ) conferred by a four-year college, 
university or professional school or an advanced degree from 
one of these institutions. 

FAMILY_STATUS 

SINGLE HEAD OF HOUSEHOLD — A single, abandoned, separated, 
divorced or widowed individual who has responsibility for one 
or more dependent children under age 18. 

RACE/ETHNIC GROUP 

WHITE (NOT HISPANIC) — A person having origins in any of the 
original peoples of Europe, North Africa, or the Middle East. 

BLACK (NOT HISPANIC) — A person having origins in any of the 
black racial groups of Africa. 

HISPANIC — A person of Mexican, Puerto Rican, Cuban, Central 
or South American, or other Spanish culture or origin 
(including Spain), regardless of race. NOTE: Among persons 
from Central and South American coun triei" nly those who are 
of Spanish origin, descent, or culture should be included in 
the Hispanic category. Persons from Brazil, Guiana, and 
Trinidad, for example, would be classified according to their 
race, and would not necessarily be included in the Hispanic 
category. Also, the Portugese should be excluded from the 
Hispanic category and should be classified according to their 



?n f^^?^'''' OR ALASKAN NATIVE - A person having origins 

in any of the original peoples of North America, and who 
maintains cultural identification through tribal affiliation 
or community recognition. 

ASIAN OR PACIFIC ISLANDER - A person having origins in any 
tL^ original peoples of the Far East, Southeast Asia, the 
Indian subcontinent (e.g., India, Pakistan, Bangladesh, Sri 
Lanka, Nepal, Sikkim, and Bhutan), or the Pacific Islands. 
This area includes, for example, China, Japan, Korea, the • 
Philippine Islands, and Samoa. Hawaiian natives are to be 
recorded as Asian or Pacific Islanders. 

OTHER BARRIERS TO EMPLOYMENT 

LIMITED ENGLISH LANGUAGE PROFICIENCY — Inability of an 
applicant, whjse native language is not English, to 
communicate in English, resulting in a job handicap. 

HANDICAPPED INDIVIDUAL - Any individual who has a physical 
or mental disability which for such individual constitutes or 
results m a substantial handicap to employment. NOTE: This 
definition includes disabled veterans for reporting purposes. 
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READING SKILLS BELOW 7th GRADE LEVEL -- An adult or youth 
assessed as having English (except in Puerto Rico) reading 
skills below the 7th grade level on a generally accepted 
standardized test. 

NOTE: The following other methods of determination may be 
used : 

° A school record of reading level determined within the 
last 12 months. 

** If an applicant is unable to read and therefore cannot 
complete a self-application for the JTPA/EDWAA program, 
s/he may be considered to have English reading skills 
below the 7th-grade level. 

Individuals with any of the following may be considered 
to have English reading skills above the 7th-grade 
level : 

- A GED certificate received within the last year. 

- A degree (usually a BA or BS) conferred by a 4-year 
college, university or professional school. 

If there is any question regarding reading ability, a 
standardized test should be administered. 

BENEFITS STATUS 

UNEMPLOYMENT COMPENSATION CLAIMANT — Any individual who has 
filed a claim and has been determined monetarily eligible for 
benefit payments under one or more State or Federal 
unemployment compensation programs, and who has not exhausted 
benefit rights or whose benefit year has not ended. 

LABOR FORCE STATUS 

UNEMPLOYED: 15 OR MORE WEEKS OF PRIOR 26 WEEKS — An 
individual who is unemployed at the time of eligibility 
determination and has been unemployed for any 15 or more of 
the 26 weeks immediately prior to such determination, has 
made specific efforts to find a job throughout the period of 
unemployment, and is not classified as "Not in Labor Force", 

VETERAN STATUS 

VETERAN — An individual who served in the active military, 
naval, or air service (of the U.S.), and who was discharged 
or released therefore under conditions other than 
dishonorable. 

NOTE: The term "active" means full-time duty in the Armed 
Forces, other than duty for training in the reserves or 
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National Guard. Any period of duty for training in the 
reserves or National Guard, including authorized travel, 
during which an individual was disabled from a disease or 
injury incurred or aggravated in the line of duty, is 
considered "active" duty. 

VIETNAM-ERA VETERAN — A veteran, any part of whose active 
military, naval, or air service occurred between August 5, 
1964 and May 7, 1975. 

PROGRAM COSTS 

ACCRUED EXPENDITURES ~ The allowable charges incurred during 
the program year to date requiring provision of funds for: 
(1) goods and other tangible property received; and (2) costs 
of services performed by employees, contractors, 
subrecipients , and other payees. 

NOTE: These charges do not include "resources on order", 
i.e., amounts for contracts, purchase orders and other 
obligations for which goods and/or services have not been 
received . 

RETRAINING ACTIVITY 

BASIC EDUCATION — Includes remedial reading, writing, 
mathematics and/or English for non-English speakers. 

OCCUPATIONAL SKILLS TRAINING — Includes vocational education 
which is designed to provide individuals with the technical 
skills and information required to perform a specific job or 
group of jobs. For reporting purposes excludes On-the-Job 
Training . 

ON-THE-JOB TRAINING — Is training in the public or private 
sector given to an individual, who has been hired first by 
the employer, while s/he is engaged in productive work which 
provides knowledge or skills essential to the full and 
adequate performance of the ]ob. 
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Example of State Follow-up Questionnaire 

New Hampshire Follow-up Questionnaire* 





*Ncw Hampshire follow-up is conducted by Employment Training Institute (ETI). 
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HELLO* MAY I SPEAK WITH 



7 (Naae on cover sheet) 



HELLO. MY NAME IS (first ind Itst). I«M TAKING A SURVEY FOR THE NEW 

HAMPSHIRE JOB TRAINING COUNCIL AND - 

...I WOULD LIKE TO ASK YOU SOME QUESTIONS ABOUT THE JTPA PROGRAM AND SEE WHAT YOU 
HAVE BEEN DOING SINCE YOU LEFT THE PROGRAM. 

(For those «ho received « letter) 

...WITHIN THE LAST WEEK OR SO. YOU SHOULD HAVE RECIEVED A LETTER EXPLAINING 
THE PURPOSE OF THIS SURVEY. 

(For DES clients) 

...ro LIKE TO ASK YOU A FEW QUESTIONS ABOUT WHAT YOU HAVE BEEN DOING SINCE 
YOU WORKED WITH THE EMPLOYMENT SERVICE IN FINDING A JOB. 

WHAT YOU SAY IS VERY IMPORTANT TO US. YOUR ANSWERS WILL BE SUMMARIZED ALONG 
WITH HUNDREDS OF OTHERS TO HELP IMPROVE NEW HAMPSHIRE'S JOB TRAINING PROGRAMS. 
CALL OF YOUR ANSWERS WILL BE KEPT CONFIDENTIAL.] 



(If respondent doesn't recognize or adalt that he/she «as enrolled In a Job 
Training Prograa. continue with Question 1.) 



(If respondent Indicates that this Is an Inconvenient tim%$ sake an appointment to 
call hWher back.) 
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SECTION 1 - IX)L REQUIRED! 

A. I WANT TO ASK YOU ABOUT TWE WEEK STARHNG ON SUNDAY, _/ / , AND ENDING ON 
SATURDAY, / /_ WHICH WAS (LAST WEEK - 2 - 3 - 4 WEEKS AGO). 



© 



DID YOU DO ANY WORK FOR PAY DURING THAT WEEK? 
Yes 

No (Go to P*rt C) 




101 



(u^ HOW MANY HOURS DID YOU WORK IHAT WEEK? 
Hours 

(T) HOW MUCH DID YOU GET PAID PER HOUR IN THAT WEEK? 

S Dollars per hour 

102 $ Total Veekly Earnings. 

(Zh) HOW MUCH EXTRA, IF ANY, DID YOU EARN IN THAT WEEK FROM TIPS, OVERTIME, 
^ BONUSES, COMMISSIONS OR ANY WORK YOU DID ON THE SIDE, BEFORE DEDUCTIONS? 

102A $ Dollars 

^' fi^L^.i^'^ ^^^^^^ ^ ^^^^^^ SUNDAY, _/_, TO 

SATURDAY # / » 

©JM^-rSi'l "^^ ^^^^ ^"^"^ '^^^^^ "0** MANY WEEKS DID YOU WORK AT 

ALL FOR PAY DURING THE 13 WEEK PERIOD? 

103 Weeks 

^' Jm..^.°v*° question 1: NOW I WANT TO ASK YOU ABOUT THE EKTIRE 13 WEEKS FROM 
SUNDAY, _/_, TO SATURDAY, / . 

DID YOU DO ANY WORK FOR PAY DURING THE 13 WEEK ..PERIOD? 

Yes (Go to Q. 5) 

No 

104 

©HOW MAY WEEKS DID YOU DO ANY WORK AT ALL FOR PAY DURING THAT 13 WEEK 
PERIOD? 

105 Weeks 

MAY I HAVE YOUR DATE OF BIRTH FOR OUR RECORDS 
106 / / 
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SECTION 2 - OPINIONS ABOUT THE JOB TRAINING PROGRAM 

2?lcuT«r4'^^ ^° QUESTIONS ABOUT YOUR EXPERIENCE WITH THE NEW 

HAMPSHIRE JOB TRAINING PROGRAM. 



o 



HOW DID YOU FIND OUT ABOUT THE PROGRAM? 

1 - Friend/ relative told me 

2 - Saw advertisement In paper 

3 - Welfare worker 

4 - School counselor 

5 - Vocational Rehabilitation Counselor 

6 - Employment Service Counselor 

7 - Other 



207 

WHAT WAS THE MAIN REASON WHY YOU ENROLLED IN THE PROGRAM? 



208 

© 



1 - To get a job 

2 - To learn a skill or trade 

3 - To get a better Job 

4 - To get a GEO (High School Equivalency Diploma) 

5 - To got off Welfare 

6 - Other 



209 



V ^To^t^Jr^ol^^^^^^ ^^'^^ "^^^^ EXPERIENCE IN THE NEW HAMPSHIRE JOB 

TRAINING PROGRAM? WOULD YOU SAY IT WAS ... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



NOW I'D LIKE YOU TO GIVE ME YOUR OPINION ABOUT SEVERAL ASPECTS OF THE PROGRAM. 

® YOU FEEL V^IS^ ^'^^^^^ APPLYING FOR AND GETTING IN THE PROGRAM. DO 
210 ^ ^ " ^"^^^ 3 - Only Fair 4 - Poor 5 - No Opinion 
(If Poor) WHY DO YOU FEEL THAT WAY? 



211 



© 



IV rl^llrJIl^ ^'^^^"^ EVALUATING YOUR SKILLS AND PREPARING A SPECIFIC 
^ TRAINING PLAN FOR YOU. DO YOU FEEL IT WAS... 

^ ■ 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 
Vl3/ (If Poor) WHY DO YOU FEEL THAT WAY? 

213 



1R3 
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©HOW ABOUT THE QUALITY OF TRAINING YOU RECEIVED (FROM THE PROGRAM/FROM 
YOUR EMPLOYER? WAS IT... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 

214 



\J WHY 



Poor) 

DO YOU FEEL THAT WAY? 



215 



®(If OJT and If worfcod during referenced week) 
ARE YOU STILL WITH THE SAME EMPLOYER? 



Yes (Go to Q. 19) 

No 



216 



U7j HOW WAS THE HELP IN FINDING A JOB AFTER YOU COMPLETED THE PROGRAM? WAS 



1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



217 



^ydf Poor) WHY DO YOU FEEL THAT WAY? 
218 



©HOW WAS THE HELP YOU RECEIVED FROM THE JOB TRAINING STAFF WHILE YOU WERE 
IN THE PROGRAM? WAS IT... 

1 - Excellent 2 - Good 3 - Only Fair 4 - Poor 5 - No Opinion 



219 



^^(If Poor) WHY DO YOU FEEL THAT WAY? 
220 



©OVERALL, WOULD YOU SAY THAT THE HELP YOU RECEIVED FROM THE JOB TRAINING 
PROGRAM HAS IMPROVED YOU ABILITY TO FIND AND KEEP A JOB? 



1 - Yes 2 - No 3 - Don't Know 



221 



(22^ WHAT WAS THE BEST PART OF THE JOB TRAINING PROGRAM IN YOUR OPINION? 



222 



If: 4 
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(5^ WHAT'S : r. )NE THING ™T YOU WOULD LIKE TO SEE CHANGED IN THE PROGRAM? 
Codes for What you would like to see changed in the program: 

1 - "Don't Know" 

2 - "Nothing" (If followed by substantive response* code it) 

3 - Better instruction/teachers 

4 - Longer courses/vrograms 

5 - Provide more financial support during training (e-g., more pay, 

better pay) 

6 - Other 

______ 

SECTION 3 - WORK HISTORY 

NOW I'D LIKE TO ASK YOU SOME QUESTIONS ABOUT ANY JOBS YOU HAVE HELD SINCE LEAVING 
THE PROGRAM, 



© 



24j (If enplcyed during reference week:) 

ARE YOU STILL EMPLOYED? 

(If une^1«ved during reference weeks) 

ARE YOU CURRENTLY WORKING? 

Yes 



No (Go to Puge 7, (L 30) 



324 



© 



25 J WHAT IS THE NAME OF THE COMPANY TWAT YOU ARE WORKING FOR? 
(If More than one enployer* 11«t Pri—ry B^>iayer) 



»25 



© 



26 J WHAT KINO OF WORK DO YOU DO? 

(Probe) WHAT ARE YOU MOST IMPORTANT DUTIES: 



326A Title: 



326B DOT Code: 



© 



27 J 00 YOU WORK FULL TIME OR PART TIME? 

327 



(28^ HOW MANY HOURS PER WEEK DO YOU WORK? (If range, give Midpoint) 

hours 

328 



ins 



Appendix C • S 



0 



29 
329 $ 



HOW MUCH ARE YOU MAKING PER HOUR? 



© 



330 



HOW MANY DIFFERENT JOBS HAVE YOU HELD SINCE LEAVING TWE PROGRAM? 
(If Zero* go to Page S$ (L 55) 



[Complete the following Job Profile for up to 3 Jobs* starting with most recent 
(not current) Job] 

MOST RECENT 

331 JOB TITLE 

332 DOT CODE 



333 START DATE 

334 END DATE 

335 DURATION 



Days - Weeks - Months 



336 HOURLY WAGE $. 



337 1 - FULL TIME / 2 - PART TIME 

338 REASON FOR LEAVING 



NEXT MOST RE(»{r 

339 JOB TITLE _ 

340 DOT CODE _ 



341 START DATE 

342 END DATE 

343 DURATION 

344 HOURLY WAGE $. 



J, 

J. 



Days - Weeks - Months 



345 1 - FULL TIME / 2 - PART TIME 

346 REASON FOR LEAVING 



NEXT MOST RECENT 

347 JOB TITLE _ 

348 DOT CODE _ 

349 START DATE _ 

350 END DATE . 

351 DURATIOK 



352 HOURLY WAGE S 

353 1 - FULL TIME / 2 - PART TIME 

354 REASON FOR LEAVING 



Days - Weeks - Months 



REASON FOR LEAVING CODES: 

1 - Laid off 

2 - Fired 

3 - Did not like work 

4 - Not enough pay 

5 - Too hard to get there (transp.) 

6 - Company went out of business 



7 - Did not like supervisor 

8 - Company moved 

9 - Decided to return to 

school /training 

10 - Found a better job 

11 - Health reasons 
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SECTION 4 - MISCELLAfCOUS 



(Ask Quastlons 55 to 59 only If currofitly uMaployod.) 



® 



ARE YOU CURRENTLY LOOKING FOR A JOB? 

Yes (Go to (L 57) 

No 



455 



© 



WHAT IS THE MAIN REASON THAT YOU ARE NOT LOOKING FOR A JOB? 

1 - Pregnant/ responsible for child care 

2 - Returning to school /training 

3 - Health 

4 - On temporary Lay-off 

5 - Moving from area 

6 - Waiting for new job to start 

7 - Have no transportation 

8 - Can't find a Job/have given up 

9 - Other 



456 (Go to (L 60) 



© 



WHAT TYPE OF JOB ARE YOU LOOKING FOR? 



457A Title: _ 
457B DOT (kxle 



0 



HOW ARE YOU GOING ABOUT LOOKING FOR A JOB? 

1 - Looking In the newspapers 

2 - Employment Service 

3 - Private employment agency 

4 - Asking friends 

5 - New Hampshire Job Training Council 

6 - Other ^ 



458 




IN LOOKING FOR A JOB, HAVE YOU BEEN ABLE TO USE THE JOB SEARCH 
TECHNIQUES OFFERED BY THE NEW HAMPSHIRE JOB TRAINING COUNCIL OR NOT? 

1 - Yes 2 - No 3 - Don't Know 



459 
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®tllllLJ;?^/TII'^?.S,^.,^?°^'- TRAINING PROGRAM SINCE LEAVING THE NEW 
HAMPSHIRE JOB TRAINING PROGRAM? 

Yes 



460 



461 



© 



® 



462 



© 



463 



464 



® 



465 



No (Go to Q- 62) 



WHAT TYPE OF SCHOOL OR TRAINING PROGRAM WAS IT? 



^iiSi S°rkl RECEIVE FINANCIAL SUPPORT FROM GOVERNMENT PROGRAMS, SUCH AS 
FOOD STAMPS, MEDICARE, UNEMPLOYMENT INSURANCE, OR WELFARE. ARE YOU 
RECEIVING ANY TYPE OF GOVERNMENT FINANCIAL SUPPORT? 

Yes 



No (Go to (L 64) 



WHAT KINDS OF ASSISTANCE ARE YOU RECEIVING? 
(Code all responses) 

1 - Food Stamps 

2 - Welfare 

3 - Unemployment Insurance 

4 - Medicaid 

5 - Day Care Assistance 

6 - Housing Subsidies 

7 - Other 



WOULD YOU RECOMMEND TWE JOB TRAINING PROGRAM TO A FRIEND OR RELATIVE? 
^ ■ Yes 2 - No 3 - Don't Know 



JHANK YOU FOR YOUR HELP ON THIS SURVEY. WE'LL BE CALLING YOU AGAIN IN 3 

1 - Yes "THANK YOU AND GOODBYE." 

2 - No (Go to (L 66) 

3 - Don't Know (Go to (L 66) 



©WOULD YOU PLEASE GIVE US A PHONE NUMBER WHERE WE WILL DEFINITELY BE 
JBLE JO^REACH YOU A FEW MONTHS FROM NOW? FOR EXAMPLE, THE PHONE NUMBER 
OF A FRIEND OR RELATIVE WHO WOULD KNOW WHERE WE COULD REACH YOU? 



466 Phone No. or 
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Sample RFP for FoIIow-Up Contractor 





*This sample RFP was adapted from RFPs used by the State of South Carolina, Budget and Control Board, 
Division of General Services; and by the Rhode Island Job Training Partnership Act Office, 
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Job Training Partnership Act Office 



Request for Proposal 

Proposal Requested 

The [State/SDA agency] seeks proposals froin qualified contractors for conducting 
a 13-week follow-up of JTPA participants. Work will include weekly telephone 
(or in person) interviews, mail questionnaire follow-up as required, data entry and 
verification, data analysis, and quarterly report preparation. The proposal must 
demonstrate that the follow-up activities meet the requirements of the U.S. 
Department of Labor Job Training Partnership Act data collection and reporting 
requirements, as specified in the Technical Assistance Guide for FoUow-Up and 
Validation (Attachment 1). Proposals will be evaluated on the basis of adherence 
to DOL requirements, bid specifications, technical approach, related experience, 
and cost. 

Response Deadline 

Proposals are to be submitted to the [State/SDA agency] no later than [time, 
date]. [Number of copies] of the proposal must be addressed to the attention of 
[name, address]. Interested bidders are invited to attend a preproposal 
conference on [time, date] at [agency, address]. Call [phone number] to confirm 
attendance or for additional information. 

Eligible Bidders 

Any governmental, non-profit, or private for-profit organization may apply. 
However, organizations that administer or operate JTTA programs serving Title 
II-A, or EDWAA participants in [State, SDA] are not eligible. 

Narrative Instructions 

The proposal narrative should explain how the bidder will provide the services 
requested in this RFP. In addition, it should demonstrate the bidder's applicable 
experience and ability to complete the survey research work required. All 
information presented in the proposal should respond clearly and concisely to the 
proposal specifications and should demonstrate that the bidder understands and is 
capable of fulfilling the responsibilities outlined in Section 6. 

Period of Performance 

The contract will be for a twelve (12) month period beginning [start date] and 
ending [end date]. The [State/SDA agency] may extend the contract if it appears 
to be in the best interest of the State/SDA and is agreeable with the Contractor. 
Such an extension may be less than, but will not exceed [maximum number]. The 
maximum contract per year is [$ amount]. 
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6. Contractor Responsibilities 



a. Select and Train Interviewers and Supervising Staff 

The contractor has sole responsibihty for selecting staff and for training 
and supervising staff in locating sample members, conducting interviews, 
recording contact attempts and results. The contractor must provide the 
[State/SDA agency] with the names and qualifications of staff assigned to 
the project, and their position titles and social security numbers. 

b. Schedule Interviews 

The contractor must develop and follow a schedule for conducting 
interviews that meets with the DOL guidelines. Interviews must be 
conducted between the 14th and 17th weeks following termination from 
JTPA. 

c. Conduct Interviews 

• Multiple attempts must be made and documented to contact each 
individual in the sample. Attempts will include telephoning the 
terminee at various times of day, telephoning the backup contacts, using 
directory assistance resources, and where necessary contacting other 
resources such as the JTPA service provider or last-known employer. 

• The contractor must make arrangements to receive return telephone 
calls on a toll-free (800 number) or collect basis from terminees in the 
sample. 

• The contractor must send letters and questionnaires to terminees in the 
sample who cannot be located by the end of the [-th week] after 
termination. 

• Interviews must be conducted directly with the terminee. Noone else 
(e.g., relatives, friends) may provide answers to the follow-up questions, 
although they may provide assistance in locating the terminee. 

d. Record and Verify Data 

The contractor will record contact attempts and responses to follow-up 
questions on the interview form following the coding scheme provided. 
The contractor will double enter the data into the MIS to ensure accuracy 
and validity. 
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Achieve Minimum Response Rates 

The contractor will achieve a 70% or better response rate for each of the 
following terminee groups: 

• Title II-A Adults in each SDA - employed at termination 

• Title n-A Adults in each SDA - not employed at termination 

• Title II-A Adult Welfare Recipients in each SDA - employed at 
termination 

• Title II-A Adult Welfare Recipients in each SDA - not employed at 
termination 

• EDWAA Participants in each SSA - employed at termination 

• EDWAA Participants in each SSA - not employed at termination 

Validate Responses to Ensure Accurate Data 

The contractor will verify that each respondent has been contacted by 
providing the respondent's date of birth. This date will be checked with 
[State/SDA] MIS records. 

In addition, the contractor will propose specific methods to ensure the 
accuracy and validity of the follow-up information collected, including 
monitoring, supervisory, and other appropriate approaches. 

Report FoIIow-Up Data 

• The contractor will report response rates for each of the groups listed in 
above on a [specify period] basis. Each [specify period] the contractor 
will transmit the follow-up data in the form of [MIS format] and submit 
the completed interview forms and any incomplete forms for individuals 
who have not mailed back responses by the end of the 18th week after 
termination. 

• At the end of each quarter, the contractor will provide summary follow- 
up reports for the [State/each SDA] for each participant category (Title 
II-A Adults, Title II-A Welfare Adults, EDWAA). Reports should 
include at a minimum the following information: 

(1) Terminees in the sample 

(2) Terminees contacted (by telephone/by mail) 
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(3) Response rates for each group ■ employed at termination and not 
employed at termination 

(4) % terminees employed at follow-up • employed at termination and 
not employed at termination 

(5) For Title II-A/JASR, average weekly earnings during 13th week (of 
employed terminees) - employed at termination and not employed 
at termination (For EDWAA, average hourly wage) 

(6) Average number of weeks worked during 13-week period - 
employed at termination and not employed at termination 

(7) [Add any other follow-up information collected by State/SDA] 

These reports will be sent to the [State/SDA-if State, also sent to each 
SDA]. 

7. [State/SDA] Responsibilities 



a. Select FoUow-Up Sample 

The [State/SDA agency] will draw random, fixed-rate samples of JTPA 
participants whose enrollment terminated each calendar week, beginning 
with [start date] and ending with [end date]. Samples will be drawn for 
Title II-A Adults, Title H-A Welfare Adults, and EDWAA terminees. 

The total weekly sample size is expected to average [#], but may vary from 
[range]. Approximately [#] of terminees will be followed-up each year. 

b. Provide Follow-Up Questionnaire 

Attachment 2 is the follow-up interview form, suggested introduction, and 
acceptable probes. 

c. Provide List of Follow-Up Sample and Contact Information 

The [State/SDA agency] will provide a computer printout listing each 
terminee in the sample, and at least two backup contacts. This information 
will be provided at least two (2) weeks before follow-up contacts with that 
terminee are to start. 

d. Provide Additional Assistance in Locating Terminees 

The [State/SDA agency] will make arrangements for JTPA service 
providers and OJT employers to assist the contractor in locating terminees 
in the sample who cannot be reached during the first two (2) weeks of the 
follow-up period. 
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e. Provide Technical Assistance 

The [State/SDA agency] will provide technical assistance to the contractor 
including the DOL follow-up requirements, training program descriptions, 
and a copy of the Technical Assistance Guide for FoUow-Up and 
Validation (Attachment 1). 

Proposal Content 

The proposal should include the following components: 

a. Procedures for completing tasks outlined in Section 6 above 

b. Capabilities of the contractor 

c. Budget with costs per contact, justified in terms of project activities and 
major expenditures. 

Selection Criteria 

Each proposal will be evaluated on the degree to which the contractor has 
addressed the specifications in this RFP, the related experience of the contractor, 
and the proposed budget. 
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How to Select a Random Sample 



The purpose of random sampling for the JTPA follow-up is to ensure that every 
terminee in each group sampled (II-A Adults, II-A Adult Welfare, III, and EDWAA) has 
an equal chance of being selected for the sample. The process of random sampling is 
straightforward, but requires care and accuracy to ensure that no bias is introduced. 

The most conventional and efficient tool for selecting a random sample is a 
random number table. A sample random number table appears in Exhibit C.l. Similar 
tables are available in most introductory statistics books and in collections of scientific 
tables. Also, many computer systems provide a routine that calculates random numbers. 

The special property of random numbers is that they are independent of «ach 
other. That is, each digit or number has an equal chance of being selected next, no 
matter what number was selected before. For example, if the first number selected is 
five, there would be an equal chance of getting any number next, incluoing a five. Using 
a random number table to select a sample ensures that the selection of one terminee is 
just as likely to occur as the selection of another. 

Several procedures may be used to select a random sample. Five procedures are 
described below. The first is automatically performed by the computer; the other four 
are manual. 

Procedure 1 • Automated MIS Selection 

Many MIS systems are capable of selecting random samples automatically. If 
these capabilities conform to generally accepted statistical practices for drawing samples, 
you can use them easily to select random samples for each group of terminees eligible 
for follow-up. 



17G 



Appendix E - 1 



Exhibit E.l 



Random Number Table 



One thousand random digits 



Colum number 



dne 
mber 


I- 


10 


11- 


20 


21 


'30 


31- 


40 


41 


-50 


1 


48453 


83321 


51097 


62267 


11801 


98016 


16292 


25447 


13088 


58586 


2 


28888 


93145 


99052 




42475 


•J w / 7 X 


34068 




1585Q 




3 


00672 


54389 


94930 


48359 


88791 


28612 


91819 

7 XO X 7 


28478 


38^21 


878^8 

O / OHO 


4 


16377 


40682 


06927 


19063 


78214 


68719 


36349 


7'*n57 
/ jyjj 1 




f,^ 1 78 
oxx / o 


5 


62923 


44829 


53414 


40567 


90322 


6QQ65 


74881 


78501 




jyjjo X 


6 


87791 


92166 


04036 


79327 


36771 


38182 


19312 


57882 


46373 


58787 


7 


84140 


76746 


92624 


16333 


69654 


31164 


09929 


00856 


53382 


92281 


8 


93051 


92816 


19703 


13278 


21939 


41750 


46815 


34755 


84802 


74573 


9 


86438 


91275 


89794 


11268 


82425 


28264 


59675 


55166 


38589 


22255 


10 


65585 


97847 


50996 


44998 


20824 


83918 


23122 


86252 


58399 


87874 


11 


79782 


96739 


40676 


51209 


01483 


69407 


37866 


85225 


15070 


84730 


12 


77028 


74590 


39227 


81418 


04191 


02715 


90163 


45956 


77515 


13251 


13 


96729 


58766 


88856 


41533 


60605 


52564 


53386 


48399 


80880 


95670 


14 


29688 


40534 


94017 


28728 


16757 


58906 


08922 


51430 


09461 


73608 


15 


06894 


81782 


73549 


30787 


00584 


13065 


91879 


69108 


58548 


20338 


16 


22155 


01471 


38289 


30201 


69312 


16013 


53975 


62224 


80687 


85778 


17 


38151 


02407 


78419 


85166 


42116 


03059 


15525 


68969 


60304 


07692 


18 


47031 


96935 


15467 


55925 


13556 


51354 


13623 


31167 


82089 


27590 


19 


62635 


35059 


95261 


39375 


75985 


51099 


78495 


76524 


49133 


29303 


20 


31630 


92345 


59647 


62634 


56886 


45408 


79105 


65756 


39667 


22319 



Appendix E - 2 



177 



Procedure 2 




Step 1: Determine the proportion of terminees to be included in the sample. For 
example if the planned number of terminees is 1,700, the minimum sample 
size is 338. The proportion of terminees included in the sample is 
338/1,700 - .199. If a decision has been made to oversample (i.e., draw 
more than 338 terminees) the proportion would be higher. 

Step 2: Determine the number of terminees to be included in the sample for the 
period for which the sample is being drawn. For example, if in a month 
there are 104 terminees, and the sampling proportion is .199, the number 
in a monthly sample would be 104 X .199 = 21 (when rounded). 

Step 3: Assign a 3- or 4- digit random number to each terminee eligible for 
follow-up. Note that the random number table is one long string of 
numbers. To select a series of 4-digit random numbers, randomly pick a 
starting point in the middle of the table. From that point, simply go 
through the list, four digits at a time. 

Step 4: Reorder the list according to the assigned random 
numbers. 

Step 5: Select as the sample the first x individuals on the list, where x is the 

desired sample size for the week or month. In the example above, the first 
21 terminees would be selected. 

Procedure 3 

Step 1: Determine the proportion of terminees in the sample as in Step 1 of 
Procedure 2. 

Step 2: Assign a 3-digit random number to each terminee eligible for follow-up for 
the week or month. 

Step 3: Choose for the sample terminees whose random number is less than the 
sampling proportion times 1,000. For example, if the sampling proportion 
is .199, choose terminees with numbers between 000 and 198~that is, 
numbers less than .199 X 1,000 = 199. 
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Procedure 4 



Step 1: Determine the proportion of terminees in the sample 
as in Step 1 of Procedure 2. 

Step 2: Use the last three digits of the terminees' social 
security numbers to identify them. 

Step 3: Choose a 3-digit random number, r. Select for the sample all terminees 
whose last three digits of their social security number are between r and r 
+ (1,000 times the sampling proportion). For example, if the random 
number is 372 and the sampling proportion is .199, take all terminees 
whose last three digits are between 372 and (372 + 199 =) 571. If the 
random number plus (1,000 times the sampling proportion) is over 999, 
take terminees whose last three digits are above the random number and 
terminees whose last three digits are below the random number plus (1,000 
times the sampling proportion) minus 1,000. For example, if the random 
number is 932, take all individuals from 932 to 999 and those from 000 to 
(932 + 199 - 1000 = ) 131. 

Procedure 5 

Step 1: Order the terminees eligible for follow-up according to the last several 
digits of their social security numbers. 

Step 2: Determine the ratio of the size of the group eligible for follow-up to the 
minimum sample size. Calculate the integer proportion for that ratio; call 
this integer n. For example, if total planned terminees is 1,700, the 
minimum sample size is 338 (from Exhibit 2.1), the ratio is 1,700/338 = 
5.02, and the integer portion is 5. 

Step 3: Randomly select a random number, r between 1 and n. 

Select for the sample the rth terminee in the ordered list and every nth 
terminee afterwards. 
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Selected Survey Research References 

This list includes two groups of references designed to guide States, SDAs, and 
contractors involved in JTPA follow-up: 

1. general survey research references ■ for guidance on overall design issues, 
methods of anaysis, and appropriate uses of the data collected. 

2. survey design and implementation references - for guidance on specific 
techniques for collecting follow-up data collection. 



General References 

Babbie, Earl R. (1973). Survey research methods. Belmont, CA: Wadsworth Publishing 
Co. 

Backstromm, Charles H. and Hursh-Cesar, Gerald (1982). Survey research . 2nd ed. 
New York: John Wiley and Sons. 

Crane, William D. and Brewer, M. B. (1985). Principles nf so cial research and 
evaluation. Boston: Allyn-Bacon. 

Isaac, Stephen and Micahel, William B. (1981). Handbook in research and evahiatinn 
2nd ed. San Diego; Edits Publishers. 



Survey Design and Implementation References 

Blankenship, A. B. (1978). PrQfes.sional telephone surveys New York; John Wiley & 
Sons. 

Department of Manpower Development, Policy and Evaluation Division (December, 
1981). Designing and implementing local follo w-up systems . Parts I and II. Boston; 
Author. 

Department of Manpower Development, Policy and Evaluation Division (June, 1980). 
Methods an d strategies to locate a n d contact survey respondents: Technical 
assistance manual. Boston: Author. 
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Dillman, Don A, (1978). Mail and telephone surveys: The total design method . 
New York: John Wiley & Sons. 

Fowler, Floyd J., Jr. and Mangione, Thomas W. (1990). Standardized Survey 
Interviewing: Minimizing Interviewer -Related Error . Newbuiy Park, CA: Sage 
Publications. 

Frey, James H. (1983). Survey research bv telephone . Beverly Hills: Sage Publicaionts. 

Lewis, Morgan V. (1972). "Finding the hard-to-locate: A review of best practices." In 
Michael E. Bonis (ed.), Evaluating t he impact of manpower programs . Uxington, 
MA: Lexington Books, D.C. Heath and Co. 

Sudman, Seymour and Bradburn, Norman M. (1982). Asking questions: A practiga ) 
guide to qu estionnaire design . San Francisco: Jossey-Bass, Inc. 

Warwick, Donald P. and Liniger, Charles A (1975). The sample survey: Thenrv and 
practice . New York: McGraw-Hill. 
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Exhibit G.l 



Worksheet to Calculate DifTerences in Response Rates 



1. Total Title II-A Adults 

a. Employed at Termination 

b. Not Employed at Termination 



I 

Size of 

Selected 

Sample 



II III 
Numbei of Response 
Completed Rate 
Interviews (coi ii + Oxioo 



c. Difference in Response Rates; 

if >5%, Go to Bias Adjustment Worksheet, and on to Part 2. 
if ^5%, Go to part 2. 



2. Title II-A Welfare Adults (Includes welfare terminees contained in Total II-A Adult 
group, and any supplementary welfare terminees) 

a. Employed at Termination I 1 I 1 I 



b. Not Employed at Termination 



c. Difference in Response Rates; 
if >5%, Go to Bias Adjustment Worksheet, and on to Part 3. 
if ^5%, Go to part 3. 



3. Dislocated Workers (EDWAA) 
a. Employed at Termination 



b. Not Employed at Termination 



c. Difference in Response Rates; 
if >5%, Go to Bias Adjustment Worksheet, 
if ^5%, Finalize caluclations. 
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Exhibit G.l (continued) 



INSTRUCTIONS: Worksheet to Calculate Differences in Response Rates 

1. In Column I (Size of Selected Sample) enter the number of terminees sampled 
for follow-up this Program Year for each group as indicated. This may be 
githcr the minimum required .-.ample determined, a larger number, or the 
entire group of eligible terminees, as determined by the State or SDA. 

2. In Column II (Number of Completed Interviews) enter the number of 
terminees with completed follow-up interviews for each sample group as 
indicated. 

3. Divide the number of completed interviews, or respondents (Column II) by the 
sample size (Column I); then, multiply the result by 100 to calculate the 
response rate. Enter the response rate in Column III (Response Rate). Each 
of these response rates should be at least 70 percent. 

4. Subtract the response rate for terminees who were not employed at 
termination (Line b) from the response rate for terminees employed at 
termination (Line a), and enter the difference on Line c for each sample as 
indicated. 

Total Title II-A Adults 

5. If the difference in response rates between II-A adults who were and were not 
employed at termination exceeds five (5) percentage points, use the 
adjustment worksheet provided (Exhibit G.2: Worksheet to Adjust FoUow-Up 
Performance Measures for Nonresponse Bias) to calculate adjusted follow-up 
performance measures for this group. 

6. If the difference in response rates is less than or equal to five (5) percentage 
points, then follow the simplified calculations contained in these instructions to 
compute the follow-up performance measures for this group. Adjusting for 
nonresponse bias is optional in this case. 

7. Proceed to Parts 2 and 3. 

II-A Adult Welfare and EDWAA/Title III Dislocated Workers: 

8. The steps for Title II-A Adult Welfare and Total Adults and EDWAA/Title 
III Dislocated Workers are the same as those listed above. 
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Exhibit G.2 presents instructions for making adjustments where the difference in 
response rates between those who were employed and those who were not employed at 
termination exceeds five (5) percentage points. Instructions involve five basic entries, 
both for terminees employed and for terminees not employed at termination, taken from 
the State's or SDA's aggregated follow-up data: 

1. The number of terp inees eligible for follow-up for the Program 
Year. (This figure > not the number entered in Column I on the 
Response Rate Worksheet in Exhibit G.l, which is the number in 
the follow-up sample. Rather it is the total number of terminees 
eligible for follow-up from which the sample was drawn.) 

2. The number of respondents (completed interviews) from the sample 
for the same period (This should correspond to the number entered 
in Column II on the Response Rate Worksheet.) 

3. The number of respondents who were employed during the 13th 
follow-up week 

4. The average gross weekly earnings for those respondents employed during the 
i3th follow-up week 

5. The average weeks worked during the 13-week follow-up period for 
all respondents 

Using these entries, you can make all of the remaining computations to adjust the follow- 
up performance measures for differences in response rates. Again, using this adjustment 
procedure is optional where the response rate differential is five percentage points or 
less. 
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Workshttt to Adjust Follow-up Pwfonnance MMsurot for Nonrosponso Bias 



Gram Radptont: 
N«nt 

AddMM 

CNy/ZIp 



ADULT A YOUTH PROGRAM 

^TWall-AAduNi 

TMtll-AAdullWtNart 

e%Adulli 

eKAduHWtNM* 



DI810CATED WORKER PROGRAMS 

SuteiiiiArMPrognn 

8M9 Prapfanw 
Swy NHofwi rm«v» 



FMowupt Conductad From 
through 



(Report Dale) 




STATUS AT TERMINATION: 



Employod 
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Exhibit G.2 (continued) 



INSTRUCTIONS: Worksheet to AcUust FoIIow-up Performance 

Measures for Non-Response Bias 

1. Check the performance group (Title II-A Adult or Adult Welfare, EDWAA) 
for which the adjustment is being made - in the upper left-hand corner. 

2. In Column A, enter the number of terminees who were employed at 
termination, not employed at termination, and the sum of the two items, for 
the group eligible for follow-up. i 

3. In Column B, enter the number of terminees, for both those employed and 
those not employed at termination, for whom completed interviews were 
obtained at follow-up. 

4. Enter the respective follow-up information for both sets of respondents in 
Columns C, E, and G, as follows: 

Column C: the number of respondents employed in 
the 13th follow-up week 

Column E: the average gross weekly earnings for those respondents 
employed in the 13th follow-up week 

Column G: the average number of weeks worked during the 13-week 
follow-up period, for all respondents 

5. To calculate the adjusted follow-up employment rate: 

- Multiply the number eligible for follow-up (Column A) by the number of 
respondents employed at follow-up (Column C) separately for those who 
entered employment and those who did not enter employment at 
termination. Then divide the two products by the number of completed 
interviews in each group (Column B). 

- Enter the result in Column D for those who were and were not employed 
at termination. 

- Add the estimated number of employed in Column D and enter the total 
in Box J. 

- Divide the estimated number of employed terminees (Box J) by the total 
number of terminees eligible for follow-up (Box I), and multiply by 100 to 
obtain the adjusted employment rate at follow-up. 

- Enter the adjusted employment rate in Box IC 
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Exhibit G.2 (continued) 



6. To calculate the adijusted average gross weekly earnings: 

- Multiply the average weekly earnings of employed respondents (Column E) 
by the estimated number of employed terminees (Column D), separately 
for those who entered and those who did not enter employment at 
termination. 

- Enter the result in Column F for both sets of terminees. 

- Sum the estimated earnings of employed terminees in Column F and enter 
the total in Box L. 

- Divide the estimated earnings of employed terminees (Be - L) by the total 
estimated number of employed terminees (Box J) to calculate the adjusted 
average gross weekly earnings of terminees employed at follow-up. 

- Enter the adjusted average weekly earnings of employed respondents in 
Box M. 

7. To calculate the adjusted average number of vi^ks worked: 

- Multiply the average number of weeks worked (Column G) by the number 
of terminees eligible for follow-up (Column A), separately for those who 
were and those who were not employed at termination. 

- Enter the result for both sets of terminees in Column H. 

- Add the estimated average number of weeks worked in the 13-week 
follow-up period and enter the result in Box N. 

- Divide the estimated total number of weeks worked in the follow-up period 
(Box N) by the total number of terminees eligible for follow-up (Box I) to 
obtain the adjusted average number of weeks worked. 

- Enter the adjusted average number of weeks worked in Box 0. 

8. Finally, take the adjusted performance measures contained in Boxes K, M and 
O~for the Employment Rate, Average Earnings, and Average Weeks 
worked~and enter these on the appropriate lines on the JASR/WAPR. 
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Software to Support JTPA Follow-up* 



Overview 

The main function of the Software to Support JTPA Follow-up is to provide: 

• a script and automatic data entry form to be used by telephone 
interviewers conducting JTPA follow-ups. I 

To support follow-up interviewing, the software also provides the capabilities for: 

• entering, or loading from a DOS file, participant information useful 
in the interview process 

• archiving the interview results to a cumulative DOS file of follow- 
up data 

• generating a JASR-Iike report, based on the archived follow-up 
interview data 

The follow-up software does not include all functions related to follow-up, such as 
maintaining a log of contact attempts; and it is written for a limited range of hardware. 
Nevertheless, the software is intended to support the interview process and facilitate the 
collection and reporting of accurate and vahd JTPA follow-up data. 

The follow-up software is contained in a high density 5-1/4" diskette, /unerican 
Institutes for Research (AIR) staff programmed the software, which is written as a 
run-time application in the Paradox2 database language. Paradox2 is published by 
Borland, Scott's Valley, California. The software can be run as a stand-alone system, on 
an IBM-compatible PC with a 80286 or 80386 processor, a hard disk, and a high density 
disk drive. (Although the software can also run on a 8088 processor, it operates quite 
slowly.) 

The follow-up software is available on request from the DOL/ETA. To obtain a 
copy, please contact Steve Aaronson and his staff at l-(202)-535-0687. 



*This software is provided by AIR, which prepared this TAG. Its use and application are not required by the 
DOL, and its availability does not constitute a DOL endorsement. 
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Loading the Software 



To load the follow-up software, follow these steps: 



1. Make a new directory called JTPACATI: mkdir JTPACATI 

2. Enter the new directory: cd JTPACATI 

3. Copy the follow-up program from the 

diskette provided onto your hard disk: copy a:*.* 

4. Type "foUowup": followup 

The follow-up software menu screen will appear. 

[Note: The software provided includes 25 sample cases, with fictitious names, social 
security numbers, demographic, and termination data. You can practice using the 
software, using these fictitious cases, then delete the cases when you are ready to enter 
your own JTPA database.] 

Tlie follow-up software, like the Paradox language in which it is written, provides 
a series of menu options. One option on the Main Menu is Directions, which 
supplements the information presented here in Appendix H. Reading these directions is 
recommended. Using the software will also familiarize you with a Paradox idiosyncracy: 
the ubiquitous use of the function key F2 to proceed to the next step in a process. When 
in doubt, Press F2. (Do not press Escape to leave the program or any menu option.) 

The main menu has the following options, which are described in turn below: 



Main Menu 




Intervlw Direction* Enter Report Archive Quit 
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Interview 

Select this menu option when you arc ready to conduct a follow-up interview. 
The first step is to select the terminee you wish to interview rext, and the system 
provides a menu for the method of selection. 

Respondent Selection Menu 



New Partial Terminee Interviewer Browse Quit 



Prior to conducting interviews, information about JTPA terminees (at least their names, 
termination statuses, and termination dates) will have been entered or loaded, using the 
Enter option on the Main Menu. All terminees will have been marked "New," indicating 
that they have not yet been interviewed for the follow-up. When they are interviewed, 
their status changes to "Complete", or to "Partial" if the interview is begun but not 
completed. 

New. Selecting New locates the first remaining "New" case and brings that 
terminee's record onto the screen. At this time, or any time during the interview, you 
can examine all of the available data on this terminee by pressing the "\" (backslash) 
key. The next step is to type the terminee's name and to call the terminee, using the 
telephone number on the screen. 

Partial. Selecting Partial locates the first remaining partially completed case for 
interview. Otherwise it acts exactly like the New option. 

Terminee. Selecting Terminee locates the first case with a terminee name 
matching the name you type in. Otherwise it acts exactly like the New option. (Be sure 
to type in the terminee's name exactly as it was entered in the database.) 

Interviewer. Selecting Interviewer locates the first non-completed case in which 
the interviewer's name matches the name you typed in. For this option, it is expected 
tliat interviewers will have been pre-assigned using the Enter option on the Main Menu. 
After locating the appropriate case, the Interviewer option acts exactly like the New 
option. (Be sure to type in the interviewer's name exactly as it was entered in the 
database.) 
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Browse. Selecting Browse shows the entire Paradox follow-up table, which can be 
viewed using the keyboard cursor keys. You can select a particular terminee to interview 
or merely examine the existing records. If you wish to interview a particular terminee, 
you should place the cursor on the row in the table for that terminee, and type ";" (a 
semi-colon). If you wish to return to the menu instead, type "!" (an exclamation point). 

Once in the interview, from whatever method of entry, you will see a series of 
screens. The software selects the proper sequence of screens, depending on the 
information given. You do not need to worry about the sequence. To exit the interview 
at any time, press F2. 

On each interview screen, three types of information appear: 

1. Background information appears in reverse capital letters. 

2. The words to be read aloud during the interview appear in usual 
upper and lower case letters. The question wordings correspond to 
the wordings in this Guide. 

3. Information to be typed on the keyboard before continuing is 
indicated with a cursor. 

Note: For follow-up questions that require a Yes or No response (#1 and #6), 
type Y or N beside the appropriate response. 

[For programmers only: The script that determines the sequence of screens is called 
"PATHWAY.SC. To change the sequence of screens, you can edit the script using a 
word processor. Before editing the sequence, be sure that the old version is safely 
copied. To implement the change you must issue the command "PD0XRUN2 
PATHWAY" after editing PATHWAY.SC. However before issuing that command, re- 
copy the file "DOLPROCS.LIB" from the original diskette.] 

Directions 

Selectmg the Direction option brings up a series of screens with explanatory 
information about the other options available in the software. Press Enter for the next 
page, PageUp for the previous page, or F2 to return to the previous menu. 
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Enter 

Selecting the Enter option creates a table of participant information that can be 
accessed and updated during the follow-up interviews. This option should be selected on 
a periodic basis (e.g., weekly) when new cases are sampled for follow-up. The 
information can either be typed in by keyboard, or loaded from an ASCII (DOS) file 
that might be exported from your agency's MIS. You have the option of entering any or 
all of the information requested, and you can specify the layout for records in the DOS 
file. The only terminee infonnation that is used to determine the sequence of follow-up 
questions is whether the termination status is "Placed" (coded "P') or not. Also, the 
termination date is used to determine the appropriate reference week(s) to refer to in 
the questions. Although other information is optional, omitting items, such as the 
participant's name, causes an awkward blank in the screen script for the interview. 

Note that accessing a case under the Enter menu causes that case to be called 
New for interview purposes. 

After selecting Enter^ you will see the following menu, whose options are 
described below: 

Enter Menu 



Save Current Restore Table Empty Load from DOS Type in Data Quit 



Save Current. Select this option to save the current follow-up table before 
proceeding. The table is saved as a dBase m file and must be given a file name with the 
SLflix ".dbf. Becausw new information entered will write over the current follow-up, select 
this option when you want to be able to retrieve the current version of the follow-up table in 
the future. 

Empty. Select this option to empty the current follow-up table before proceeding. 
Be sure to save existing data (using the Save Current option) before using Empty. If you 
merely want to add more information about 'the terminees (in the same sort order!), then it 
is not appropriate to empty the cunent follow-up table before proceeding. In all other 
cases, it is. (Use Empty when you are ready to remove the 25 fictitious cases provided with 
the software.) 
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The use of the dBase III file format for temporary storage of follow-up tables allows 
users with dBase III but not Paradox to make flexible use of the interview results. The 
system is oriented to handle follow-up interviews in batches (such as weekly batches). The 
Archive function (see page H-7), on the other hand, allows you to aggregate the results of 
multiple follow-ups in DOS files; and the Report function (see page H-6) makes use of the 
archived DOS files. 

Restore Table. Select this option to restore a saved follow-up table before 
proceeding. The table is assumed saved as a dBase III file with suffix ".dbf . 

Load from DOS. Select this option if you wish to load terminee information from an 
existing DOS file. If you choose this option, you will be asked to examine and alter the 
specifications of columns and lengths for various fields. If a particular field is not on the 
file, you should skip that specification. After making changes and pressing F2, you will be 
asked to enter the name of the DOS participant file that is the source of information. After 
entering the file name, the information will be loaded into the follow-up table. You can 
then add more information manually by selecting Enter and not emptying the table before 
proceeding to the T^pe in Data option. 

T^pe in Data. Select this option to enter or change terminee data. In keying in data, 
you can repeat the value of the same field from the preceding case by typing (carat). 

The following fields provide terminee information: 



Participant 
Name 

Telephone Number 
Address, City, State, Zip 
Social Security Number 
Termination Status, Date 
Termination Employer, Wage 
Gender, Race, Age, 
High school diploma? 
Welfare recipient? 



Other 

Other Contact Name 

Other Contact Phone 
Training Program Name 

Interviewing Agency Name 
Interviewer's Name 

Mail Notice Date 
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Report 



By selecting Report, you can acquire the summary follow-up information needed for 
the JASR report. An ASCII (DOS) file created by the Archive function may accumulate 
results of batches over a prolonged period, such as quarterly or annually. After you specify 
the ASCII file, the follow-up software computes the summary statistics (sample size, 
response rate, follow-up employLtient rate, follow-up earnings, and follow-up weeks worked) 
for all cases and for cases that were welfare recipients. To comply with DOL regulations, 
the software automatically calculates the follow-up statistics and adjusts for the response- 
rate differential between participants placed and those not placed at termination. 

[For programmers only: The computations can be reviewed by examining the script in the 
DOS file named "O.SC, the essential part of which is shown below.] 



Script for JASR Follow-up Computations 



afs-afsp+af sn wf s-wfsp+wfsn 

if (afs) >0 then arr-round((100*(arrp+arrn)/(afsp+afsn)) ,1) 

else arr--l 
endlf 

if (wfs) >0 then wrr-round((100*(wrrp+wrrn)/(wfsp+wfsn)) ,1) 

else wrr--l 
endif 

afer-round(100*(afsp*aep /arrp+afsn*aen /arrn)/ 

(afsp*arrp/arrp+afsn*arrn/arrn) , 1) 
wfer-round(100*(wfsp*wep /wrrp+wfsn*wen /wrrn)/ 

(wfsp*wrrp/wrrp+wfsn*wrrn/wrrn) , 1) 
af te-round( (af sp*atep/aep+af sn*aten/aen) / 

(afsp*aep/aep +afsn*aen/aen ),0) 
wf te-round( (wf sp*wtep/wep+wf sn*wten/wen) / 

(wfsp*wep/wep +wfsn*wen/wen ),0) 
afww-round( ( af sp*awwp/arrp+af sn*awwn/arrn) / 

(afsp*arrp/arrp+af sn*arrn/arrn) , 1) 
wfww-round( (wf sp*wwwp/wrrp+wf sn*wwwn/wrrn) / 

(wfsp*wrrp/wrrp+wfsn*wrrn/wrm) , 1) 

Notes: The prefix "a" refers to all adults, "w" to welfare recipients. 

The suffix "p" refers to placed terminees, "n" to those not placed, 
"fs" is follow-up sample; "rr" is number or percentage of respondents; 
"fer" is follow-up emplo3anent rate; "fte" is follow-up earnings; and 
"fww" is follow-up weeks worked. Division by zero yields zero. 
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Note that no distinction is made in this system between JTPA Title II-A and 
EDWAA. The system can be implemented on separate copies of the follow-up software for 
the different titles. 



Archive 

Selecting Archive on the Main Menu allows you to save the results of the follow-up 
by adding information to a specifiable DOS (ASCH) file. The follow-up software makes use 
of DOS ASCII files because the system is designed for an environment in which other 
software may be used for information management and reporting. After selecting Archive, 
type in the name of the DOS file to which information is to be added. Then select J (for 
JASR) or N (for Notes) to indicate where the information should be stored. Two kinds of 
information are stored on different files for use with other software: 

1. For JASR reporting purposes, the follow-up responses are stored, along 
with the terminee's SSN, the date of the interview, and the interviewer's 
name. The only pre-follow-up information stored on this file is the 
information needed for the JASR computations: the termination status ("P" 
or other) and whether the terminee had been a Welfare Recipient at 
eligibility determination ("Y" or "N"). 

2. For other uses, a file is written with the terminee's SSN and three fields, 
on three successive records: "Notes" (a 240-byte field containing any 
information keyed by the interviewer, such as finding an error in 
termination information); "Problems" (a 240-byte field containing the 
terminee's response (if asked) to a question about problems obtaining 
employment); and "Advice" (a 240-byte field containing the terminee's 
response to a request for advice to improve the training program in the 
future). 

In order for you to access the Archive option repeatedly for status updates, selecting the 
Archive function will not alter the condition of the current follow-up table. However, the 
information is added to the end of the DOS file named, so you must be careful not to add 
multiple copies of the information to the archive file. For example, you may delete or 
rename an existing archive DOS file before archiving another. 
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Quit 

Selecting the Quit option exits the follow-up software program and returns you to 
DOS. All files remain intact, unless the hard disk crashes. Exiting from the follow-up 
software in any other way (e.g., by turning of the power) may lose some data. 



Appendix H - 9 



